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FOREWORD 


On  October  3,  1995,  Dr.  Paul  Kaminski,  the  Under  Secretary  of  Defense  for  Ac¬ 
quisition  and  Technology,  announced  that  he  was  “requiring  that  the  Simulation,  Test, 
and  Evaluation  Process-let’s  call  it  STEP— shall  be  an  integral  part  of  our  Test  and 
Evaluation  Master  Plans.  This  means  our  underlying  approach  will  be  to  model  first, 
simulate,  then  test,  and  then  iterate  the  test  results  back  into  the  model.  ’’  He  emphasized, 
“Just  as  we  speak  now  of  ‘test,  fix,  test’...  we  should  now  plan  our  development  programs 
so  that  they  ‘model,  test,  model.  ’  My  intent  is  to  ensure  modeling  and  simulation  truly 
becomes  an  integral  part  of  our  test  and  evaluation  planning.  We  must  consider  all  the 
tools  and  sources  of  information  available  to  us  in  developing  and  evaluating  the  per¬ 
formance  of  our  weapon  systems.  ’’ 

The  Simulation  Test  and  Evaluation  Process  is  one  that  integrates  both  simulation 
and  test  for  the  purpose  of  interactively  evaluating  and  improving  the  design,  perform¬ 
ance,  joint  military  worth,  survivability,  suitability,  and  effectiveness  of  systems  to  be  ac¬ 
quired  and  improving  how  those  systems  will  be  used.  STEP  significantly  reengineers  the 
way  modeling  and  simulation  is  used  with  test  and  evaluation  to  support  acquisition  re¬ 
form.  STEP  implements  the  intent  of  the  DoD  5000-series  acquisition  regulations  and  is 
anticipated  to  contribute  to: 

n  Substantially  reducing  the  time,  resources,  and  risk  associated  with  the  acquisition 
process, 

D  Increasing  the  quality,  military  utility,  and  supportability  of  systems  developed  and 
fielded,  while  reducing  their  total  ownership  costs,  and 

D  Facilitating  Integrated  Product  and  Process  Development  (IPPD)  across  the  full  ac¬ 
quisition  life  cycle. 

We  are  providing  these  STEP  Guidelines  for  all  those  who  are  involved  with  the 
acquisition,  fielding,  and  employment  of  new  military  capabilities.  We  expect  engineers, 
users,  support  planners,  trainers,  managers,  and  executives  to  use  the  process  and  benefit 
from  it.  Within  T&E,  our  intent  is  to  ensure  that  modeling  and  simulation  truly  becomes 
an  integral  part  of  our  test  and  evaluation  process.  We  will  expect  to  see  this  reflected  in 
test  planning  and  conduct  and  documented  in  Test  and  Evaluation  Master  Plans. 


Patricia  Sanders 
Director,  Test,  Systems 
Engineering  and  Evaluation 
OUSD(A&T) 
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Philip  E.  Coyle 
Director 

Operational  Test  and  Evaluation 
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CHAPTER  1 


Introduction  to  the  STEP  Concept 


1.1  Introduction 

The  Department  of  Defense  is  seeking  to  streamline  ways  in  which  it  acquires 
weapons  systems.  Evolving  modeling  and  simulation  tools  have  the  potential  to  allow  us 
to  reduce  the  time,  resources,  and  risk  associated  with  the  process  while  improving  the 
quality  of  the  systems  produced  through  a  strategy  called  Simulation  Based  Acquisition 
(SBA). 

1.2  Acquisition  Reform  and  Integrated  Product  and  Process  Development 


The  DoD  is  committed  to  reforming  its  acquisition  system  to  achieve  essential  ef¬ 
ficiencies.  The  Department  is  developing  better  ways  to  determine  what  to  buy  and  better 
methods  of  buying  what  it  needs.  To  determine  what  it  will  buy,  the  DoD  is  placing  con¬ 
siderable  emphasis  on  selecting  the  most  cost-effective  mix  of  individual  systems  for  de¬ 
velopment  and  fielding.  In  order  to  improve  how  it  buys,  the  DoD  has  directed: 

“The  PM  shall  employ  the  concept  of  Integrated  Product  and  Process  Develop¬ 
ment  (TPPD)  throughout  the  program  design  process  to  the  maximum  extent  prac¬ 
ticable.  The  use  of  Integrated  Product  Teams  (IPTs)  is  a  key  tenet  of  IPPD...The 
IPPD  management  process  shall  integrate  all  activities  from  product  concept 
through  production  and  field  support,  using  multi-disciplinary  teams  to  simultane¬ 
ously  optimize  the  product  and  its  manufacturing  and  supportability  to  meet  cost 
and  performance  objectives.  It  is  critical  that  the  processes  used  to  manage,  de¬ 
velop,  manufacture,  verify,  test,  deploy,  operate,  support,  train  people,  and  even¬ 
tually  dispose  of  the  system  be  considered  during  program  design.”* 

Modeling  and  simulation  (M&S)  supports  the  IPPD  process  and  the  integration  of 
complex  systems  and  is  a  key  tool  of  the  Integrated  Product  Teams(IPT).  The  members 
of  an  IPT  (i.e.,  those  involved  in  design  engineering,  test,  manufacturing,  logistics,  prod¬ 
uct  support)  share  information  and  data  from  tests  and  simulations  and  identify  needed 
information  from  these  simulations  and  tests.  Furthermore,  technical  and  operational 
challenges,  which  can  be  identified  early  in  system  development  through  simulation,  can 
be  targeted  for  further  testing.  Virtual  prototypes  embedded  in  realistic  synthetic  envi¬ 
ronments  can  aid  in  developing  a  shared  vision  of  the  proposed  system  and  provide  a 
means  for  understanding  the  complex  interactions  among  the  configuration  items  in  the 
system  design.  Design,  manufacturing,  and  test  engineers  can  work  together  in  IPTs  to 


’  DoD  5000.2-R,  Part  4.2. 
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build  a  prototype  that  can  be  more  efficiently  manufactured  and  tested.  This  efficient  use 
of  accredited  M&S  is  required  by  DoD  acquisition  regulations; 

“Accredited  modeling  and  simulation  shall  be  applied,  as  appropriate,  throughout 
the  system  life-cycle  in  support  of  the  various  acquisition  activities:  requirements 
definition;  program  management;  design  and  engineering;  efficient  test  planning; 
result  prediction;  and  to  supplement  actual  test  and  evaluation;  manufacturing; 
and  logistics  support.  PMs  shall  integrate  the  use  of  modeling  and  simulation 
within  program  planning  activities,  plan  for  life-cycle  application,  support,  and 
reuse  models  and  simulations,  and  integrate  modeling  and  simulation  across  the 
functional  disciplines.”^ 

In  addition  to  increasing  the  effectiveness  of  the  design,  test,  and  manufacturing 
functional  specialists,  modeling  and  simulation  will  benefit  the  product  support  members 
of  the  team  (e.g.,  the  logisticians  and  maintainers)  as  well  as  the  training  and  warfighting 
communities. 

Program  offices  need  to  support  and  use  modeling  and  simulation  more  than  ever 
before  and  must  plan  for  the  funding  of  program  and  legacy  M&S.  Modeling  and  simu¬ 
lation  capability  has  matured  to  the  point  where  it  can  serve  as  a  key  facilitator  of:  (1)  de¬ 
velopment;  (2)  communication  between  government  and  contractor;  (3)  requirements  ex¬ 
ploration  in  the  context  of  cost  as  an  independent  variable;  (4)  demonstrating  the 
significance  of  features  found  in  component  and  subcomponent  tests;  (5)  test  planning 
and  analysis;  (6)  communication  between  engineering,  manufacturer,  tester  and  user; 
(7)  logistics  management;  and  (8)  training  and  human  factors  evaluation  during  the  life- 
cycle  of  a  system.  In  other  words,  modeling  and  simulation  used  well  in  the  IPTs  can  be 
a  key  contributor  to  the  Integrated  Product  and  Process  Development  (IPPD)  that  the  Sec¬ 
retary  of  Defense  has  directed. 


1.3  Simulation  Based  Acquisition  (SB A) 


The  Department’s  vision  is  to  have  an  acquisition  process  that  is  enabled  by  the 
robust,  collaborative  use  of  simulation  technology  that  is  integrated  across  acquisition 
phases  and  programs.  The  goals  of  Simulation  Based  Acquisition  are  to: 

•  Substantially  reduce  the  time,  resources,  and  risk  associated  with  the  acquisition  process; 

•  Increase  the  quality,  military  utility,  and  supportability  of  fielded  systems  while  reducing 
total  ownership  costs,  and 
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•  Enable  Integrated  Product  and  Process  Development  (IPPD)  across  the  full  acquisition 
life  cycle. 

SBA  is  an  integrator  of  simulation  tools  and  technology  across  acquisition  func¬ 
tions  and  program  phases  and  across  programs.  It  is  a  concept  in  which  M&S  as  a  re¬ 
source  is  more  efficiently  managed  in  the  acquisition  process.  In  a  defense  environment 
of  decreased  funding,  SBA  addresses  both  the  decreasing  availability  of  resources  for 
system  development  and  the  increasing  power  of  M&S  tools. 

1.4  The  Simulation,  Test  and  Evaluation  Process  -  STEP  Concept 


The  Simulation,  Test  and  Evaluation  Process  (STEP)  is  a  major  DoD  initiative 
designed  to  improve  the  acquisition  process  by  integrating  M&S  with  T&E.  STEP  is 
consistent  with  the  regulations  that  govern  systems  acquisition  and  does  not  require  their 
modification. 

STEP  is  a  move  beyond  the  “test,  fix,  tesf’  approach  to  a  “model-simulate-fix- 
test-iterate  approach”  with  problems  fixed  as  they  are  discovered.  This  approach,  illus¬ 
trated  in  Figure  1-1,  (model  first;  simulate;  test;  fixing  after  each  step  and  then  iterate  the 
test  results  back  into  the  model)  is  reiterated  throughout  system  development.  There  are 
many  iterative  loops  in  this  process,  for  instance,  one  can  model,  simulate,  fix,  simulate, 
fix,  simulate,  fix,  test,  then  feed  the  results  into  the  model.  When  a  need  to  fix  is  discov¬ 
ered,  the  time  for  each  fix  can  be  much  shorter  when  the  fix  can  be  verified  in  the  model 
in  hours  or  days,  as  opposed  to  a  field  test  which  can  take  weeks  or  months  to  verify  a  fix. 


1 - - - — - 1 
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Figure  1-1.  STEP  Uses  aModel-Simulate-Fix-Test-Iterate  Approach 

With  STEP,  the  set  of  models  matures,  culminating  in  representations  of  the  sys¬ 
tem,  its  interfaces,  and  its  environment  with  an  established  fidelity.  When  tests  are  con- 
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ducted,  the  data  collected  while  evaluating  the  system  can  be  used  to  refine  and  validate 
the  models.  These  models  and  simulations  can  then  be  reused  throughout  the  weapon 
system’s  life  cycle  to  further  predict  and  extrapolate  performance,  operational  effective¬ 
ness,  suitability,  and  survivability.  They  will  also  be  available  for  later  modifications  to 
the  weapon  system  and  to  other  programs. 

The  STEP  emphasis  is  the  interdependent  manner  in  which  simulation,  engineer¬ 
ing,  management,  and  test  are  applied  and  remain  available  for  reuse  throughout  the  sys¬ 
tem  life  cycle.  Credible  representations  of  the  system  and  simulations  can  provide  early 
and  continuous  insight  and  projections  and  predictions  about  system  performance;  risk 
and  risk  mitigation;  operational  effectiveness,  survivability,  and  suitability;  and  to  support 
others  in  the  acquisition,  requirements,  cost  analysis,  training,  and  user  communities. 

The  T&E  community  will  work  with  the  program  office  through  the  program  IPTs 
to  develop  a  comprehensive  evaluation  strategy  using  STEP  and  other  means.  In  order  to 
implement  this  strategy,  the  program  office  must  invest  early  to  insure  that  valid  models 
and  simulations  are  available  and  useable  when  needed  by  the  contractor  and  the  rest  of 
the  IPT. 


Figure  1-2.  STEP  is  an  Evaluation  Process 


The  program  office  can  use  the  information  in  the  following  chapters  to  assist  in 
developing  and  implementing  an  evaluation  strategy  that  will  enhance  both  their  program 
and  other  acquisition  activities. 
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CHAPTER  2 


STEP  Strategy 


2.1  Definition  and  Overview 


The  Simulation  Test  and  Evaluation  Process  is  defined  as  an 
iterative  process  that  integrates  simulation  and  test  for  the  purpose  of 
interactively  evaluating  and  improving  the  design,  performance,  joint 
military  worth,  survivability,  suitability,  and  effectiveness  of  systems 
to  be  acquired  and  improving  how  those  systems  are  used. 

In  STEP,  simulation  and  test  are  integrated,  each  one  depending  on  the  other  to  be 
effective  and  efficient.  Simulations  provide  predictions  of  the  system’s  performance  and 
effectiveness,  while  tests  are  “part  of  a  strategy  to  provide  information  regarding  risk  and 
risk  mitigation,  to  provide  empirical  data  to  validate  models^  and  simulations,  to  permit 
an  assessment  of  the  attainment  of  technical  performance  specification  and  system  ma¬ 
turity,  and  to  determine  whether  systems  are  operationally  effective,  suitable,  and  surviv- 
able  for  intended  use.”'^  A  byproduct  of  this  process  is  a  set  of  models  and  simulations 
with  a  known  degree  of  credibility  providing  the  potential  for  reuse  in  other  efforts. 

The  product  of  STEP  is  information  that  can  be  used  throughout  the  system  life 
cycle.  By  beginning  the  process  early,  models  and  simulations  can  provide  information 
on  failure  modes  during  &e  early  stages  of  design.  This  allows  for  design  iterations  be¬ 
fore  hardware  is  available  for  test.  In  addition,  throughout  the  life  cycle,  information 
from  STEP  can  support  decisions  regarding  cost-performance  trade-offs,  technical  risk, 
system  maturity,  operational  effectiveness,  suitability,  survivability,  training,  mission 
planning,  and  tactical  employment.  The  STEP  process  does  not  end  with  system  field¬ 
ing  and  deployment;  it  continues  throughout  the  operational  support  of  the  system  to  the 
end  of  its  life  cycle.  STEP  results  in  an  efficiently  designed  and  thoroughly  understood 
system  with  known  operational  effectiveness,  suitability,  and  survivability  characteris¬ 
tics. 

The  implications  of  STEP  for  DoD  acquisition  are  numerous.  Decision-makers 
can  use  system  performance  predictions  to  assess  the  military  worth  of  the  system  not 
only  before  any  physical  prototypes  are  built  but  also  throughout  the  system  life  cycle. 
The  development  and  T&E  communities  can  use  predictive  simulations  to  evaluate  sys- 


^  Model,  n  A  physics,  mathematical,  or  otherwise  logical  representation  of  a  system,  entity,  phenomenon, 
or  process,  v  To  create  a  representation  of  a  system,  entity,  phenomenon  or  process. 

Simulation.  A  method  for  implementing  a  model  over  time. 
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tern  performance  in  areas  that  are  not  readily  testable,  or  to  encourage  testing  a  system  to 
failure  to  improve  reliability.  The  developmental,  operational,  and  live  fire  T&E  com¬ 
munities  can  use  STEP  as  a  mechanism  for  test  planning,  for  sharing  data  for  models, 
simulations,  and  tests,  and  for  establishing  a  compatible  and  consistent  basis  from  which 
to  evaluate  performance. 

STEP  is  a  sub-process  of  the  overall  acquisition  process  and  applies  to  all  acqui¬ 
sition  programs,  not  only  Major  Defense  Acquisition  Programs  (MDAP)  and  Major 
Automated  Information  Systems  (MAIS). 

A  key  ingredient  to  successfully  accomplishing  STEP  is  the  development  of  an 
interdependent  and  consistent  set  of  STEP  resources.  STEP  resources  are  both  simula¬ 
tion  and  test  tools  that  allow  the  analyst  and  tester  to  focus  on  that  which  is  essential  to 
evaluate,  to  monitor  the  activities  as  they  occur,  and  to  consolidate  and  analyze  the  re¬ 
sults  of  their  activities.  Testing  tools  include  live  tests,  stimulators,  and  laboratory  fa¬ 
cilities  that  have  supported  testing  for  many  years.  Simulation  resources  include  a  set  of 
models  that  together  describe  the  system  characteristics  and  performance  at  all  levels 
from  engineering  models  to  campaign  level  wargames.  They  are  used  in  a  variety  of 
ways  from  measuring  compliance  to  design  requirements  through  predicting  system  per¬ 
formance  in  an  operational  environment.  Available  STEP  resources  are  defined  further 
in  Chapter  4. 


2.2  STEP  Throughout  a  Weapons  Systems  Life  Cycle  Process 


As  an  iterative  process  extending  across  a  weapon  system’s  life  cycle,  STEP  is 
integral  to  an  acquisition  strategy,  interacting  with  other  functions  in  each  phase  to  pro¬ 
vide  information  needed  for  acquisition  decisions  and  to  ensure  that  improvements  are 
worked  back  into  the  functional  areas  after  each  iteration.  The  involvement  of  STEP 
throughout  the  acquisition  process  is  shown  in  Figure  2-2.  The  figure  also  depicts  the 
developing  models  and  simulations,  products  of  STEP,  that  are  available  for  continued 
use  throughout  the  life  cycle  of  the  system.  As  STEP  progresses  across  the  acquisition 
phases,  some  models  and  simulations  developed  for  specific  applications  can  be  modi¬ 
fied,  further  verified  and  validated,  and  made  ready  for  use  later  in  the  acquisition  proc¬ 
ess. 


During  Analysis  of  Mission  Need.  Models  and  simulations  offer  a  way  to 
quantify  the  shortfalls  in  number  and  type  of  systems  for  force  structure  analysis,  to 
quantify  the  consequences  for  a  wide  variety  of  scenarios,  and  to  identify  thresholds  of 
operational  significance  to  the  accomplishment  of  national  military  objectives. 
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During  Concept  Exploration.  During  this  phase,  an  Analysis  of  Alternatives 
(AOA)  is  developed  to  aid  and  document  decisionmaking  by  providing  insight  into  the 
relative  advantages  and  disadvantages  of  the  options  being  considered  to  meet  a  mission 
need.  Analyses  performed  show  the  sensitivity  of  each  alternative  to  changes  in  key  as¬ 
sumptions  (e.g.,  threat)  or  major  variables  (e.g.,  speed,  probability  of  kill,  weight).  In  ad¬ 
dition,  the  most  promising  system  concepts  are  defined  in  terms  of  broad  objectives  for 
cost,  schedule,  performance,  trade-off  opportunities,  overall  acquisition  strategy,  and 
T&E  strategy. 

In  implementing  STEP,  programs  should  address  major  information  objectives, 
simulation  and  test  events,  resources,  and  timing.  This  includes  determining  how  the  test 
objectives  will  be  met  and  developing  plans  for  the  logical  progression  of  events  to  gather 
the  appropriate  data.  These  plans  will  integrate  the  needs  of  the  T&E  community  with 
those  of  other  acquisition  activities  (e.g.,  design,  risk  management,  trade-off  studies)  and 
the  training  community.  When  implemented,  they  should  result  in  the  identification  of  an 
initial  set  of  interactive  models  that  provide  continually  maturing  and  comparable  results 
to  support  program  decisions. 


During  Program  Definition  and  Risk  Reduction.  During  the  Program  Defini¬ 
tion  and  Risk  Reduction  (PDRR)  phase,  STEP  can  provide:  early  insight  into  the  reliabil¬ 
ity,  availability,  and  maintainability  (RAM)  of  the  proposed  system;  information  to  sup¬ 
port  the  assessment  of  risk;  information  from  ergonomic  models  to  support 
maintainability;  information  on  physics  of  failure;  and  data  on  the  human-machine  inter¬ 
face. 


Predictions  from  engineering-level  M&S  are  also  used  as  the  basis  for  represent¬ 
ing  the  system’s  performance  in  engagement-level  M&S  (i.e.,  the  predictions  replace  the 
assumed  values).  Engagement-level  M&S  provides  information  on  the  effectiveness  of 
the  system  against  a  specific  target  or  enemy  threat,  which  can  be  used  to  support  MOE 
projections  at  the  system-on-system  level. 

Early  Operational  Assessments  (EGAs)  in  support  of  Milestone  n  cannot  always 
be  performed  on  physical  prototypes,  which  are  often  difficult  to  acquire  at  this  point  in 
the  program.  STEP  will  provide  the  program  manager  with  an  integrated,  credible  set  of 
models  that  can  be  used  in  simulated  test  scenarios  to  allow  completion  of  EGAs  when 
physical  prototypes  are  not  available. 
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During  Engineering  and  Manufacturing  Development.  During  the  Engineer¬ 
ing  and  Manufacturing  Development  (EMD)  phase,  the  most  promising  design  approach 
is  translated  into  a  stable,  interoperable,  producible,  supportable,  and  cost-effective  de¬ 
sign.  This  is  when  models,  simulations  and  tests  are  used  to;  (1)  verify  the  system’s  de¬ 
sign;  (2)  confirm  that  design  risks  have  been  controlled;  (3)  certify  readiness  for  opera¬ 
tional  testing;  and  (4)  evaluate  the  system’s  operational  effectiveness,  suitability,  and 
survivability.  The  use  of  models  that  have  been  updated  with  test  data  increases  the  un¬ 
derstanding  of  the  military  worth  of  the  system  design. 

Feedback  from  tests  to  the  simulations  is  not  only  required  for  verification,  vali¬ 
dation,  and  accreditation  (W&A),  but  also  to  allow  progressive  improvement  in  the 
M&S.  In  this  manner  M&S  can  support  the  tests  with  performance  predictions  for  use  in 
planning  future  tests  and  in  risk  assessment.  Feedback  can  also  highlight  areas  where  ad¬ 
ditional  M&S  and  test  effort  may  be  needed,  e.g.,  gaps  in  requirements  testing,  refinement 
of  environmental  constraints,  and  adjustments  to  modeling  logic  or  algorithms.  The  final 
result,  when  testing  is  complete  and  the  final  M&S  updates  have  been  made,  will  be  ma¬ 
ture,  validated  M&S  resources,  which  will  support  the  weapon  system  throughout  its  life 
cycle. 


During  Production,  Fielding/Deployment,  and  Operational  Support.  The 

process  of  updating  models  and  data  sets  is  critical  in  resolving  design  problems  after 
fielding  and  in  making  modifications  to  the  system  throughout  its  life  cycle.  During  the 
last  phase  of  the  acquisition  process,  the  entire  validated,  interdependent  set  of  STEP  re¬ 
sources  is  available  for  use.  Changes  introduced  during  production  must  also  be  made  in 
models  and  data  sets  to  ensure  they  remain  representative  and  documentation  is  complete. 

STEP  can  be  useful  through  the  entire  service  life  of  the  system.  It  can  aid  in  the 
development  of  training  simulators  with  the  requisite  realism  and  fidelity  for  system  op¬ 
erators  and  maintainers.  Operating  specifications  and  characteristics  in  manuals  and 
electronic  mission  planning  aids  can  also  be  derived  from  M&S,  i.e.  projections  about 
maximum  range.  Records  of  system  usage,  failures,  and  maintenance  actions  are  some¬ 
times  incorporated  into  models  of  system  performance  and  longevity,  permitting  the  revi¬ 
sion  of  operating  procedures  and  the  timing  of  periodic  depot  maintenance  to  reflect  ac¬ 
tual  in-service  behavior.  Authoritative  system  representations,  developed  and  matured 
throughout  the  acquisition  process,  can  be  applied  in  training  exercises  from  the  mission 
to  the  campaign  level. 

Figure  2-2.  STEP  in  the  Acquisition  Process 
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2.3  Relationship  between  STEP  and  the  T&E  Process 

The  DoD  T&E  Process^  is  a  five  activity  iterative  process  that  is  repeated  many 
times  throughout  the  acquisition  life-cycle.  Figure  2-3  shows  a  flow  chart  of  one  iteration 
of  the  T&E  process  with  the  five  major  activities,  and  inputs  and  outputs  to  other  re¬ 
search,  development,  and  acquisition  activities. 

A  misperception  is  that  STEP  uses  M&S  to  support  the  T&E  process,  that  using 
STEP  is  business  as  usual.  In  fact,  STEP  is  revolutionary  in  the  ways  it  extends  beyond 
the  T&E  stovepipe  to  use  information  from  across  the  community  to  contribute  to  the 
evaluation.  STEP  enhances  the  T&E  process,  as  well  as  the  other  acquisition  processes, 
and  provides  a  mission-centered  focus  on  evaluation. 

Another  misperception  is  that  STEP  uses  M&S  to  replace  testing.  STEP  allows 
the  T&E  community  to  better  focus  the  tests  that  are  done  to  get  critical  information  for 
the  evaluation.  Through  IPPD,  early  involvement  of  the  entire  T&E  community  with  the 
requirements,  acquisition,  and  training  communities  will  help  ensure  that  the  mission 
need  and  the  system  are  fully  understood  and  described  in  operational  terms. 

STEP  enhances  the  T&E  process  with  the  application  of  M&S  tools.  In  addition 
to  the  collection  of  data  to  support  system  evaluation,  data  are  collected  to  refine  the 
models  and  simulations  using  the  model-test-model  approach.  Testing  produces  models 
and  simulations  with  increased  credibility,  and  allows  for  the  assessment  of  system  per¬ 
formance  in  areas  and  under  conditions  that  might  not  be  otherwise  available  with  con¬ 
ventional  testing  methods. 

STEP  integrates  modeling  and  simulation  into  the  T&E  process  by  applying 
analysis  and  other  resources  that  support  the  development  of  an  integrated,  comprehen¬ 
sive,  and  flexible  evaluation  strategy  early  in  the  acquisition  cycle  to  insure  continual, 
comprehensive  performance  evaluations.  These  performance  evaluations  serve  not  only 
the  T&E  community  and  the  acquisition  community  during  the  acquisition  process,  but 
also  the  requirements  community  prior  to  milestone  zero,  and  the  training  community 
throughout  the  system’s  life. 

STEP  contributes  to  refining  the  requirements  in  the  Operational  Requirements 
Document.  Sensitivity  analyses  using  STEP  models  and  simulations  identify  perform¬ 
ance  drivers  and  aid  in  determining  many  of  the  operational  issues  required  by  the  pro¬ 
gram  such  as:  critical  operational  issues  (COIs),  critical  technical  issues,  and  critical  sys¬ 
tem  interfaces  that  must  be  evaluated.  STEP  also  helps  identify  evaluation  measures — 
MOEs,  MOPs,  and  CTPs — and  where  appropriate,  the  associated  threshold  and  objective 
criteria.  STEP  can  also  be  used  to  analyze  data  collected  to  support  the  performance 
evaluation.  This  solid  foundation  leads  to  the  creation  of  a  comprehensive  evaluation 
plan  with  clearly  defined  data  and  resource  requirements,  which  supports  an  affordable 
acquisition  strategy  and  a  realistic  program  schedule. 

^  The  generally  accepted  DoD  T&E  process  is  documented  in  “A  Description  of  the  DoD  Test  and  Evalua¬ 
tion  Process  for  Electronic  Warfare  Systems”. 


2-7 


Models  and  simulations  can  be  used  to  focus  tests  on  critical  missions  by  identi¬ 
fying  the  high  payoff  scenarios  from  the  spectrum  of  possible  mission  scenarios.  Also, 
weapon  system  immaturity  can  be  partially  mitigated  by  increasing  crew  training  using 
appropriate  human-in-the-loop  simulators  prior  to  field  testing,  thereby  avoiding  wasted 
time  while  operators  learn  how  to  exploit  the  new  weapon  system’s  unique  capabilities 
and  develop  effective  tactics. 

The  wise  application  of  STEP  will  produce  objective  and  more  comprehensive  in¬ 
formation  regarding  performance  than  the  use  of  T&E  alone.  M&S  can  free  costly  test 
assets  for  use  in  defining  the  edges  of  operating  envelopes  where  confidence  may  be  low 
but  performance  is  expected  to  be  high.  Simulations  can  also  be  used  to  test  to  failure  in 
order  to  improve  reliability.  This  evaluation  strategy  applies  the  elements  of  the  T&E 
process  to  gain  knowledge  and  understanding  in  order  to  produce  better  products  without 
detracting  from  the  test  and  evaluation  process.  This  kind  of  evaluation  strategy  provides 
better  performance  information  to  the  warfighters  and  wisely  allocates  costly  assets  to 
evaluate  the  most  appropriate  performance  issues. 
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CHAPTER  3 
STEP  Implementation 


3.1  Introduction 


Based  upon  a  thorough  understanding  of  simulation  and  test  capabilities  and 
limitations,  the  program  office  must  obtain  the  most  cost-effective,  objective,  and  credi¬ 
ble  performance  information  possible  to  adequately  evaluate  the  system  under  develop¬ 
ment  and  its  performance  in  the  intended  operational  environment.  Implementation  of 
STEP  places  the  emphasis  on  an  overall  evaluation  strategy  that  uses  all  available  data 
sources  efficiently  and  effectively. 

STEP  helps  explore  the  best  sources  of  data  to  support  analyses  and  to  provide 
performance  information  in  time  to  support  decision-makers.  Archived  data  from  train¬ 
ing  exercises,  war  games,  contractors,  government  agencies,  studies,  after-action  reports, 
and  patrol  reports  are  all  potential  sources  of  data.  Data  from  models,  simulations,  and 
test  events  may  also  be  used  to  augment  archived  data  that  is  not  applicable,  available, 
certified,  or  complete.  In  some  cases  this  may  require  modifications  to  existing  data  col¬ 
lection  methods  or  analysis  assets.  In  other  cases,  new  instrumentation  and  new  analysis 
capabilities  may  be  needed. 

The  important  ingredients  in  successful  implementation  of  STEP  are  (1)  a 
thoughtful  and  thorough  evaluation  plan  that  covers  the  entire  life  cycle  process,  (2) 
early  identification  of  all  the  tools  and  resources  needed  to  execute  that  evaluation  plan 
and  timely  investment  in  those  resources,  (3)  assuring  the  credibility  of  the  tools  to  be 
employed,  and  (4)  once  testing  is  accomplished,  using  the  resulting  data  to  improve  the 
efficacy  of  the  models  and  simulations.  The  evaluation  methodology  outlined  in  the 
TEMP  should  incorporate  the  STEP  process. 


3.2  Planning  for  an  Evaluation  Strategy 


The  first  key  is  to  determine  the  life-cycle  evaluation  strategy.  This  means  identi¬ 
fying  performance  drivers  and  COI  using  M&S  at  the  campaign,  mission,  and  engage¬ 
ment  levels  to  help  determine  what  mission  critical  issues  must  be  evaluated  and  what 
methods  should  be  used  to  make  those  evaluations. 

Planning  begins  at  a  macro  level  and  gradually  moves  to  the  micro  level.  STEP  is 
a  natural  part  of  the  planning  process  that  begins  with  an  analysis  of  operational  mission 
needs  and  proceeds  through  the  selection  of  the  most  appropriate  sources  of  data  (micro 
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level)  to  provide  credible  performance  information.  Planners  develop  a  flexible  evalua¬ 
tion  strategy,  which  can  be  modified  if  necessary,  to  support  the  eventual  fielding. 

The  challenge  in  developing  a  comprehensive  evaluation  strategy  is  identifying 
the  appropriate  data  sources  and  the  associated  analysis  methods  that  will  provide  the 
most  credible  information  in  time  for  each  of  the  information  users. 

STEP  might  use  validated  models  and  simulations  to  identify  performance  ex¬ 
pectations  in  the  center  of  operating  envelopes  and  use  valuable  test  assets  to  define  the 
edges  of  those  envelopes.  In  other  cases,  models  and  simulations  will  be  used  to  make 
performance  predictions  that  will  be  validated  through  actual  test  events.  In  still  other 
cases,  extrapolated  results  from  validated  models  and  simulations  will  be  used  when  ca¬ 
pabilities  cannot  yet  be  tested.  The  T&E  community  will  provide  technical  expertise  re¬ 
garding  the  best  method  of  evaluating  the  mission  capabilities  required.  This  truly  moves 
the  T&E  process  to  the  place  where  it  provides  information  and  understanding  of  joint 
military  worth  to  the  warfighters. 

Engineers  and  operational  evaluators  play  a  key  role  in  evaluating  system  per¬ 
formance  throughout  the  life  cycle.  They  identify: 

•  The  information  needed  and  when  it  must  be  available.  This  includes  understanding 
the  performance  drivers  and  the  critical  issues  to  be  resolved. 

•  The  exact  priority  for  what  must  be  modeled  first,  then  simulated,  and  then  tested. 
This  includes  learning  about  the  subcomponent  level,  the  components,  and  the  system 
level. 

•  The  analysis  method  to  be  used  for  each  issue  to  be  resolved.  Timing  may  have  a  sig¬ 
nificant  effect  on  this.  The  design  function  can  use  models  before  any  hardware  is 
available.  It  will  always  be  more  expedient  to  use  models  and  simulations  at  the  early 
stage  of  system  design.  However,  the  design  itself  may  be  affected  by  operational 
considerations  that  require  examination  of  real  tests  or  exercise  data.  It  will,  given  the 
training  and  logistic  information  required  of  systems  today,  be  prudent  in  the  long  run 
to  develop  appropriate  models  and  simulations. 

•  The  data  requirements  and  format  for  the  analysis  chosen.  Included  in  this  determi¬ 
nation  is  the  availability  of  instrumentation,  not  only  for  collecting  performance  data, 
but  also  for  validating  appropriate  models  and  simulations. 


Evaluation  Strategy 

The  evaluation  strategy  considers  the  time,  risk,  cost,  type,  and  quality  of  infor¬ 
mation  needed  and  identifies  the  analysis  methods  and  resources  that  can  provide  the  data 
required.  Planners  then  trade  off  the  sources  of  data  to  ensure  use  of  the  most  cost- 
effective  set  of  STEP  resources.  Current  trends  suggest  M&S  will  play  a  more  important 
role  in  providing  needed  data.  Leaders  in  the  acquisition  and  T&E  communities  must 
ensure  needed  models  and  simulations  are  developed  and  refined  as  the  system  under  de- 
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velopment  matures.  Furthermore,  these  leaders  must  commit  to  providing  the  resources 
needed  to  develop  and  refine  the  models  and  simulations. 

Testing  usually  provides  highly  credible  data,  but  safety,  environmental,  and  other 
constraints  can  limit  operational  realism,  and  range  cost  and  scheduling  can  be  limiting 
factors.  Modeling,  especially  credible  model  building,  may  be  very  expensive  although 
M&S  can  be  available  before  hardware  is  ready  to  test.  A  prudent  mix  of  simulation  and 
testing  is  needed  to  ensure  that  some  redesign  is  possible  (based  on  M&S)  before  manu¬ 
facturing  begins.  In  the  PPD  context,  modeling  should  iteratively  examine  the  design 
within  the  mission  context  and  in  the  combat-simulated  environment.  The  system  com¬ 
ponents  that  contribute  significantly  to  mission  performance  require  a  greater  degree  of 
testing  than  those  whose  function  is  not  as  closely  tied  to  mission  performance. 

The  type  of  system  being  developed  will  influence  the  evaluation  strategy,  and 
availability  of  M&S  resources.  For  example,  developers  of  strategic  systems  have  long 
made  extensive  use  of  models  and  simulations  because  testing  was  impractical;  thus, 
many  of  the  resources  they  need  already  exist.  On  the  other  hand,  other  communities 
have  not  made  such  extensive  use  of  M&S  and  may  have  to  develop  new  models.  For 
less  complex  systems,  STEP  may  reveal  that  test  is  the  most  cost-effective  means  to 
evaluate  system  performance,  and  the  development  of  models  and  simulations  is  not  war¬ 
ranted. 


Operational  requirements  (including  threat  projections)  and  the  information  re¬ 
quired  to  support  acquisition  decisions  influence  the  evaluation  strategy.  Timing  is  criti¬ 
cal.  The  acquisition  schedule  must  provide  time  for  engineers,  analysts,  and  decision¬ 
makers  to  gather  the  data  and  information  they  need.  At  the  campaign  and  mission  levels, 
information  will  be  directed  toward  resolving  macroscopic  issues  on  performance,  techni¬ 
cal  risk,  schedule,  and  cost.  Realistic  scenarios  using  models  and  simulations  and  based 
upon  MOEs  soundly  supported  by  MOPs  and  CTPs  will  address  these  issues  in  terms  of 
campaign  outcomes  and  mission  success. 

Models  and  simulations  can  be  used  for  sensitivity  analyses  to  identify  perform¬ 
ance  drivers,  to  obtain  a  good  understanding  of  the  COIs,  and  to  derive  the  CTPs.  Meth¬ 
ods  for  measuring  CTPs,  MOPs,  and  MOEs  and  analyzing  the  collected  data  to  resolve 
critical  issues  must  be  selected.  The  analysis  methods  and  the  available  analysis  re¬ 
sources  can  have  a  significant  impact  on  the  data  that  needs  to  be  collected.  In  the  case  of 
new  technologies,  if  the  existing  M&S  cannot  address  the  critical  issues,  then  the  modifi¬ 
cation  of  existing  or  the  development  of  new  analysis  resources,  instrumentation,  and  data 
collection  methods  may  be  considered. 

Risk  reduction  must  be  a  key  consideration  when  developing  the  evaluation  strat¬ 
egy.  Operational  challenges  to  the  technology  must  be  clearly  understood.  The  early 
evaluation  must  focus  on  those  areas  of  the  system  under  development  that  present  the 
greatest  technical  risk.  The  evaluation  may  involve  refining  existing  or  developing  new 
models  or  simulations.  For  example,  planners  may  want  to  provide  for  an  HWIL  facility 
as  early  as  possible  for  a  new  seeker  technology.  This  may  require  modification  or  devel- 
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opment  of  new  STEP  resources  to  provide  the  facilities,  people,  and  other  support  needed 
to  evaluate  the  seeker  technology  using  actual  hardware  and  software  with  simulated  en¬ 
vironments  and  threats.  Field  tests  may  be  able  to  accomplish  the  same  goals  before  the 
information  is  required,  while  models  and  simulations  being  developed  may  not  be  ready 
in  time  to  provide  the  information  needed  to  support  the  decision.  Also,  M&S  will  not 
identify  “unknown  unknowns”;  laboratory  or  field  tests  are  needed  to  accomplish  this. 

Software  development  has  been  shown  to  be  an  area  of  considerable  technical 
risk.  When  considering  the  technical  risk  and  timing  required  for  having  analyses  avail¬ 
able  to  support  the  information  needs  of  a  decision-maker,  contingency  or  alternative  op¬ 
tions  should  be  considered.  The  evaluation  itself  should  contain  alternatives  to  be  im¬ 
plemented  if  the  STEP  resources  of  choice  are  not  available.  Such  alternatives  should  be 
thoroughly  considered  to  ensure  they  represent  the  next  best  method  of  providing  consis¬ 
tent,  comparable,  and  credible  performance  information. 

The  bottom  line  is  that  early  and  thorough  planning  for  the  evaluation  strategy  is 
essential  for  effective  implementation  of  STEP. 


3.3  Resource  Identification  and  Investment 


Program  managers  and  STEP  implementers  must  begin  early  to  plan,  budget,  and 
integrate  interactive  M&S  in  design,  development,  acquisition,  testing,  training,  and  lo¬ 
gistic  efforts,  starting  with  currently  available  models  used  by  the  contractor  and  the  gov¬ 
ernment.  A  key  element  of  an  improved  acquisition  system  is  the  development  and  matu¬ 
ration  of  an  interdependent,  consistent  set  of  models  and  simulations  using  test  data. 

While  developing  the  evaluation  strategy,  the  program  office  must  also  develop  a 
plan  to  identify  and  fund  resources  that  support  the  evaluation.  In  determining  the  best 
source  of  data  to  support  analyses,  STEP  considers  credibility  and  cost.  Resources  for 
simulations  and  test  events  are  weighed  against  desired  confidence  levels  and  the  limita¬ 
tions  of  both  the  resources  and  the  analysis  methods.  The  program  manager  works  with 
the  T&E  IPT  to  use  STEP  to  develop  a  comprehensive  evaluation  strategy  that  uses  data 
from  the  most  cost-effective  sources;  this  may  be  a  combination  of  archived,  simulation, 
and  test  event  data,  each  one  contributing  to  addressing  the  issues  for  which  it  is  best 
suited. 


STEP  integrates  the  use  of  simulation  events  with  test  events,  each  depending  on 
the  other,  to  add  value  to  the  T&E  process.  Simulations  are  part  of  a  strategy  to  provide 
information  by  predicting  test  results,  by  expioring  performance  in  the  testable  realm,  and 
by  extrapolating  performance  in  the  non-testable  realm.  Tests  provide  information  from 
the  real  world  regarding  situations  and  environments  and  permit  an  assessment  of  the  at¬ 
tainment  of  technical  performance  specifications  and  system  maturity.  Tests  are  used  to 
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determine  whether  systems  are  operationally  effective,  suitable,  lethal,  and  survivable  in 
combat. 

The  reader  will  find  a  discussion  of  the  types  of  STEP  resources  that  can  be  ap¬ 
plied  to  the  evaluation  strategy  in  Chapter  4. 

Identification  of  Models  and  Simulations 

For  new  systems,  planning  should  begin  early  enough  to  influence  the  initial  Re¬ 
quest  For  Proposal  (RFP).  If  government  campaign-,  mission-,  and  engagement-level 
models,  simulations,  and  architecture  are  specified  in  the  RFP,  industry  can  respond  with 
a  proposal  that  includes  the  contractor’s  plan  for  the  development  and  use  of  engineering- 
level  models,  simulations,  that  are  interoperable  with  Government  models,  and  the  asso¬ 
ciated  preliminary  evaluation  plan.  Such  content  should  be  a  maj  or  proposal  evaluation 
criterion.  Contractor  models  would  interface  with  the  models  used  by  the  government 
and  specified  in  the  RFP.  Thus,  common  models  and  simulations  would  be  identified  and 
available  at  source  selection,  through  EMD,  and  throughout  the  life  cycle.  Further,  ap¬ 
propriate  models  developed  by  the  contractor  would  be  documented,  delivered,  and 
maintained  to  support  this  integrated  evaluation  process. 

Many  models  may  not  be  developed  by  the  acquisition  program  office.  Cam¬ 
paign-level  and  mission-level  models  and  simulations  may  be  the  property  of  other  or¬ 
ganizations  or  agencies  and  in  some  cases  may  be  owned  or  at  least  operated  by  industry. 
Major  facilities  may  be  owned  by  other  Services.  Supported  by  top-level  managers,  plan¬ 
ners  from  key  organizations  should  develop  an  integrated  approach  that  specifies  analysis 
resources,  methods,  assumptions,  types  of  data,  and  the  data  sharing  to  take  place  that 
uses  both  the  resources  and  facilities  specific  to  the  program  and  other  available  re¬ 
sources. 

The  first  consideration  when  using  M&S  is  to  conduct  an  in-depth  analysis  to  de¬ 
fine  what  the  M&S  is  required  to  do.  Before  any  decisions  are  made  about  applying 
M&S  to  a  given  problem,  the  problem  itself  must  be  defined  and  articulated  clearly 
enough  to  permit  a  precise  specification  of  where  M&S  will  play  a  role  in  the  solution  of 
the  problem,  and  how  it  will  contribute  to  the  solution.  Only  then  can  acceptance  criteria 
be  developed  for  candidate  M&S. 

It  is  important  to  define  what  M&,S  will  be  required  to  do  (i.e.,  functionality),  and 
it  is  also  necessary  to  determine  how  well  candidate  M&S  must  do  those  things. 

Functional  Requirements.  Functional  requirements  are  “nonanalytical”  re¬ 
quirements  in  the  sense  that  they  do  not  contribute  directly  to  the  resolution  of  program 
decisions.  Instead,  they  define  such  factors  as  hardware  and  software  compatibility  re¬ 
quirements  (e.g.,  the  M&S  must  mn  on  a  certain  type  of  workstation  under  a  certain  oper¬ 
ating  system);  pre-  and  post-processing  requirements  for  M&S  data  (e.g.,  M&S  inputs  or 
outputs  must  be  converted  to  special  file  formats);  and  operations  and  training  support 
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requirements  (e.g.,  M&S  cannot  have  license  agreement  or  operator  training  requirements 
because  there  is  no  budget  for  such  items,  or  no  time  for  training). 


3.4  M&S  Credibility 


An  integral  part  of  planning  is  determining  the  limitations  and  the  degree  of  credi¬ 
bility  of  the  information  derived  from  use  of  the  models  and  simulations.  Furthermore,  it 
is  essential  to  plan  to  collect  real  test  data  with  which  to  validate  the  models  and  simula¬ 
tions  as  the  system  under  development  matures.  In  some  cases,  STEP  may  require  the 
collection  of  test  event  data  for  the  sole  purpose  of  validating  models  or  simulations  in 
order  to  adhere  to  the  established  evaluation  strategy. 

Many  of  the  models  needed  to  support  STEP  already  support  the  systems  acqui¬ 
sition  process,  both  in  DoD  and  industry.  Often  these  models  have  been  developed  for  a 
specific  function  or  purpose  with  little  thought  to  their  integration  into  the  entire  acquisi¬ 
tion  process.  As  dramatic  advances  in  supporting  technologies  make  these  models  and 
simulations  more  powerful  and  less  expensive,  and  as  declining  resources  and  changing 
priorities  make  it  essential  to  find  better  ways  to  develop  and  field  new  systems,  the  use 
of  M&S  and  of  associated  improved  processes  that  exploit  their  contribution  is  expand¬ 
ing. 

Validated  simulation  input  data  is  essential  for  the  use  of  accurate  and  appropriate 
M&S  resources  in  STEP.  Data  repositories  are  not  yet  populated  for  most  M&S  re¬ 
sources.  However,  limited  validated  data  for  environmental  factors,  human  interaction, 
and  some  system  performance  parameters  do  exist.  It  is  important  that  the  source  of 
M&S  input  data  be  verified  and  validated.  Questionable  data  can  introduce  unnecessary 
risk  into  the  program  and  obviate  test  data. 

Success  with  STEP  does  not  come  easy,  nor  is  it  free.  STEP,  by  integrating  M&S 
with  testing,  provides  additional  sources  of  early  data  and  alternative  analysis  methods, 
not  generally  available  in  tests  by  themselves.  It  seeks  the  total  integration  of  STEP  re¬ 
sources  to  optimize  the  evaluation  of  military  worth  throughout  the  life  cycle.  The  central 
elements  of  STEP  are:  the  acquisition  of  information  that  is  credible;  avoiding  duplica¬ 
tion  throughout  the  life  cycle;  and  the  reuse  of  data,  tools,  and  information. 
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The  success  of  STEP  depends  greatly  upon  the  credibility  of  the  STEP  resources  and 
data  which  are  employed  throughout  the  process.  Ensuring  that  the  models,  simula¬ 
tions,  and  data  used  in,  or  generated  by,  STEP  have  been  properly  accredited  and  certi¬ 
fied  must  become  a  rigid  discipline.  This  section  briefly  describes  veriGcation,  valida¬ 
tion,  and  accreditation  (W&A)  and  veriGcation,  validation,  and  cerUGcation  (W&C), 
the  requirement  for  careful  software  conGguraUon  management,  and  some  of  the 
practical  issues  involved  in  the  use  ofM&S  tools.^ 

VeriGcaGon,  Validation,  Accreditation,  and  CertiGcation 

An  effective  evaluation  strategy  must  include  iterative  Verification  and  Validation 
(V&V)  of  models  and  simulations  as  the  system  design  matures  in  order  to  provide  credi¬ 
ble  information.^  The  credibility  of  the  M&S  and  supporting  data  is  measured  by  a 
structured  V&V  process.  Based  upon  adequate  V&V,  the  M&S  is  approved  as  acceptable 
for  use  in  a  particular  application  by  accreditation,  with  the  entire  process  known  as 
W&A.  Similarly,  data  used  in  the  M&S  are  certified  through  a  process  called  W&C.* 
Ensuring  that  M&S  and  data  used  in  STEP  have  been  properly  accredited  and  certified 
must  become  a  rigid  discipline. 

The  STEP  approach  involves  integrating  the  data  from  laboratory  and  field  tests  to 
validate  models  and  simulations  as  the  system  matures.  This  will  result  in  a  more  com¬ 
plete  set  of  models  and  simulations  with  a  fidelity  that  gains  credibility  as  the  system  un¬ 
der  development  matures.  The  V&V  aspect  of  developing  models  and  simulations  may 
be  difficult,  time-consuming  and  costly,  but  it  is  also  absolutely  necessary  for  credibility 
of  the  entire  process.  W&A  is  so  important  to  the  credibility  of  models  and  simulations 
that  all  Services  require  specific  W&A  procedures  and  documentation.  These  STEP 
guidelines  are  not  intended  to  specify  W&A  procedures,  however  Annex  C  provides  ref¬ 
erences  to  some  additional  information. 

Verification  focuses  on  M&S  capahiiity  while  validation  focuses  on  M&S  credi¬ 
bility.  Verification  is  the  process  of  determining  that  model  implementation  accurately 
represents  the  developer’s  conceptual  description  and  specifications.  Validation  is  the 
process  of  determining  the  degree  to  which  a  model  is  a  sufficiently  accurate  representa¬ 
tion  of  the  real  world  (the  subject  system  and  the  operating  environment)  from  the  per¬ 
spective  of  the  intended  uses  of  the  model. 


®  The  DoD  Verification,  Validation  and  Accreditation  (VV&A)  Recommended  Practices  Guide  and 
DoDI  5000.61  should  be  reviewed  for  more  detail  on  this  subject. 

^  Verification.  The  process  of  determining  that  model  or  simulation  implementation  accurately  represents 
the  developer’s  conceptual  description  and  specification.  Verification  also  evaluates  the  extent  to  which  the 
model  or  simulation  has  been  developed  using  sound  and  established  software  engineering  techniques. 

Validation.  The  process  of  determining  the  degree  to  which  a  model  or  simulation  is  an  accurate 
representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model  or  simulation. 

Accreditation.  The  official  certification  that  a  model  or  simulation  is  acceptable  for  use  for  a  specific 
purpose. 

*  Certification.  The  determination  that  data  have  been  verified  and  validated,  and  are  acceptable  for  use  in 
the  specific  application  identified. 

Also  see  definitions  for  Data  Validation,  Data  Verification,  Data  Certification,  and  Data  VV&C. 
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Accreditation  reflects  a  decision  to  use  an  M&S  tool  for  a  specific  purpose  or  ap¬ 
plication.  The  decision  is  supported  by  the  V&V  and  certain  documentation.  The  proc¬ 
ess  that  leads  up  to  an  accreditation  decision  gathers  all  the  information  about  specific 
model  or  simulation  capabilities  relative  to  the  requirements  of  a  specific  application. 
This  information  includes  the  V&V  results,  but  also  includes  such  information  as  simula¬ 
tion  run  time,  number  of  simulation  operators  required,  the  simulation’s  history  of  use, 
documentation  status,  and  configuration  management  records.  Documentation  includes 
the  V&V  plan,  the  accreditation  plan,  and  associated  reports. 

All  M&S  are  driven  by  data,  either  as  direct  inputs  by  the  user  or  as  embedded 
constants  that  drive  simulation  characteristics.  Both  the  data  producer  and  the  data  user 
are  involved  in  data  verification,  ensuring  that  data  meet  specified  constraints  defined  by 
data  standards,  and  that  the  data  are  transformed  and  formatted  properly.  Likewise,  both 
are  involved  in  data  validation,  assessing  whether  the  data  are  appropriate  for  use  in  the 
intended  model  within  stated  criteria  and  assumptions.  There  are  data  V&V  processes 
and  procedures  which  parallel  those  for  M&S  V&V  described  previously.  For  data,  the 
decision  to  use  data  for  a  specific  application  lies  behind  the  certification.  The  configu¬ 
ration  management,  documentation,  and  similar  issues  relevant  to  M&S  V&V  also  un¬ 
derlie  data  W&C. 

Software  configuration  management  (CM)  is  of  critical  importance  to  STEP.  CM 
is  a  development  life  cycle  process  through  which  the  integrity  and  continuity  of  software 
development,  upgrades,  and  maintenance  are  recorded,  communicated,  and  controlled. 
Without  effective  CM,  a  user  cannot  be  assured  of  what  version  of  the  M&S  an  applica¬ 
tion  is  using  or  what  code,  hardware,  and/or  data  is  being  used.  Good  CM  usually  implies 
good  documentation.  Poor  CM  leaves  any  M&S  documentation  suspect  in  terms  of  cur¬ 
rency,  content,  or  both. 

The  reader  is  referred  to  the  DoD  W&A  Recommended  Practices  Guide  for  more 
information  regarding  the  W&A  process. 


Practical  Considerations 

The  process  for  selecting  V&V  tasks  rationally  within  a  constrained  budget  envi¬ 
ronment  revolves  around  the  need  for  M&S  and  data  credibility  balanced  by  concern  for 
the  cost  of  the  V&V  activities  that  contribute  to  it.  Also,  while  W&A  enhances  a  simu¬ 
lation’s  credibility  it  cannot  guarantee  that  the  M&S  results  will  be  correct,  that  the  re¬ 
sults  will  be  correctly  analyzed  and  interpreted,  or  that  the  correct  model  was  chosen  to 
solve  the  problem.  A  manager  must  be  confident  of  the  value  added  by  the  W&A  and 
W&C  process. 

Although  it  is  generally  possible  to  specify  the  kind  of  V&V  needed  to  support  a 
given  level  of  credibility,  the  amount  of  V&V  required  to  establish  credibility  for  a  par¬ 
ticular  application  will  still  be  dependent  on  a  clear  definition  of  how  program  decisions 
are  affected  by  M&S  outputs. 
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Ongoing  W&A  activities  are  the  price  one  should  expect  to  pay  for  ascertaining 
and  maintaining  the  credibility  of  the  M&S  tools.  When  a  simulation  is  modified,  it  is 
usually  done  with  the  intent  of  improving  its  operation,  accuracy  or  scope.  These  changes 
may  affect  the  simulation’s  suitability  for  particular  applications.  The  changes  must  be 
compared  against  the  user’s  intent  (verification),  and  the  impact  of  the  changes  on  simu¬ 
lation  output  must  also  be  compared  against  the  real-world  system  or  process  to  measure 
the  increase  or  decrease  in  fidelity  (validation).  Additionally,  when  the  real  world 
changes,  or  the  M&S  is  used  for  a  purpose  different  from  that  originally  intended,  previ¬ 
ous  W&A  results  should  be  reviewed  to  determine  the  impact  of  these  changes  on  the 
credibility  of  the  simulation.  Since  the  real  world  is  rarely  static  over  any  length  of  time, 
it  is  useful  to  review  the  W&A  status  of  an  M&S  periodically  to  ensure  consistency  with 
the  real  world. 


3.5  Enabling  legacy  value  of  M&S  through  STEP 


When  planning  for  resources  to  support  the  evaluation  strategy,  the  system  devel¬ 
oper  will  need  to  identify  resources  to  mature  the  models  and  simulations,  and  plan  for 
the  test  data  to  verify  and  validate  the  performance  of  those  M&S.  This  is  a  critical  to 
insuring  that  M&S  used  by  the  system  have  a  long  term  legacy  value  and  is  fundamental 
to  the  reuse  focus  of  STEP. 

The  M&S  verified  and  validated  during  development  and  testing  have  wide  use 
during  fielding  and  deployment.  Training  simulators  for  system  operators  and  maintain- 
ers  must  produce  as  realistic  an  environment  and  reproduction  of  system  functions  as  pos¬ 
sible.  This  will  most  likely  be  based  upon  or  made  up  of  subsets  of  the  earlier  M&S  re¬ 
sources.  Some  operating  specifications  and  characteristics  presented  in  manuals  and 
electronic  mission  planning  aids  are  also  derived  from  such  M&S  assets.  Models  of  sys¬ 
tem  durability  and  longevity  will  be  used  in  determining  timing  of  servicing,  planning  for 
parts  availability,  and  other  logistical  support  requirements.  Records  of  system  usage, 
failures  and  maintenance  actions  are  folded  into  certain  models  of  system  performance 
and  longevity.  This  allows  necessary  adjustments  to  operating  procedures  and  limita¬ 
tions,  plus  periodic  depot  maintenance  timing  and  planned  work,  to  meet  actual  in-service 
behavior.  Data  sets  or  M&S  are  also  rolled  up  as  increasingly  simplified,  but  authentic, 
representations  of  the  system  in  wargame  exercises  from  the  mission  to  the  campaign 
level. 


The  process  of  updating  models  and  data  sets  is  important  in  resolving  design 
problems  after  fielding  and  in  making  modifications  to  the  system  throughout  its  life  cy¬ 
cle.  Changes  introduced  during  production  must  also  be  made  in  models  and  data  sets  to 
ensure  they  remain  representative. 

The  M&S  verified  and  validated  during  development  and  testing  have  wide  use 
during  fielding  and  deployment.  Training  simulators  for  system  operators  and  maintain- 
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ers  must  produce  as  realistic  an  environment  and  reproduction  of  system  representations 
as  possible. 

Test  design  should  allow  for  data  collection  to  update  models  and  to  support  the 
continuous  V&V  process  to  ensure  that  the  most  credible  M&S  resources  will  be  avail¬ 
able  for  use  for  future  program  endeavors,  as  well  as  by  other  program  offices  and  acqui¬ 
sition  functions. 

3,6  Documenting  STEP  in  the  Test  and  Evaluation  Master  Plan  (TEMP) 


“A  TEMP  shall  provide  a  road  map  for  integrated  simulation,  test,  and  evaluation 
plans,  schedules,  and  resource  requirements  necessary  to  accomplish  the  test  and  evalua¬ 
tion  program.”®  Above  all,  STEP  should  increase  the  focus  in  the  TEMP  on  developing 
and  documenting  a  robust  and  comprehensive  evaluation  strategy.  Additionally,  the 
TEMP  addresses  how  credible  data  will  be  obtained  to  support  the  evaluation  strategy  by 
integrating  the  use  of  models  and  simulations  with  test  events.  The  TEMP  remains  a 
“living”  document  that  must  respond  to,  and  reflect  changes  in,  the  state  of  the  program 
and  its  previous  testing  results,  requirements,  models  and  simulations,  analyses,  and  ac¬ 
quisition  strategy,  schedule,  and  funding. 

The  TEMP  documents  a  comprehensive  evaluation  strategy  that  integrates  the  use 
of  M&S  with  test  events  through  STEP.  The  integration  of  STEP  into  the  individual 
parts  of  the  TEMP  is  described  below. 


Parti  System  Introduction 

Part  I  of  the  TEMP  addresses  the  evaluation  measures  and  criteria  derived  from 
system  requirements  to  be  used  in  the  evaluation  strategy.  Evaluation  measures  include 
operational  Measures  of  Effectiveness  (MOEs),  and  supporting  Measures  of  Performance 
(MOPs)  and  Critical  Technical  Parameters  (CTPs),  derived  from  operational  require¬ 
ments.  Measures  of  effectiveness  include  measures  of  suitability  and,  where  appropriate, 
measures  of  lethality  and  survivability  (susceptibility  and  vulnerability). 

Prior  to  drafting  the  TEMP,  M&S  tools  should  help  identify  performance  drivers 
to  aid  in  defining  Critical  Operational  Issues  (COIs),  MOEs,  MOPs,  CTPs,  and  critical 
system  interfaces  that  must  be  evaluated.  These  same  M&S  tools  should  also  help  deter¬ 
mine  the  quantifiable  criteria  (i.e.,  appropriate  thresholds  and  objectives)  for  these  meas¬ 
urable  parameters.  This  solid  foundation  leads  to  a  comprehensive  evaluation  strategy 
with  clearly  defined,  measurable  parameters  and  the  associated  quantifiable  criteria  that 
will  be  recorded  in  Part  I  of  the  TEMP. 


®DoD5000.2-R,  Part  3.4.11. 
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The  ability  to  trace  evaluation  results  to  requirements  is  vital  to  demonstrating  the 
degree  to  which  the  system  performance  satisfies  the  requirements.  The  iterative  nature 
of  STEP  permits  linking  operational  performance  measures  and  the  derived  technical  pa¬ 
rameters  back  to  the  operational  requirements.  Evaluation  measures  may  be  specified  in 
the  MNS,  the  ORD,  or  the  RFP,  or  they  may  be  developed  based  upon  COIs  identified 
using  M&S  tools. 

Part  II Integrated  Test  Program  Summary 

Part  n  addresses  the  schedule  and  management  of  the  models,  simulations,  test 
events,  resources,  time,  and  funding  to  carry  out  the  evaluation  strategy.  STEP  affects  the 
integrated  schedule  by  calling  for  M&S  throughout  the  evaluation  process. 

Key  development  or  modification  milestones  of  any  simulations  needed  to  support 
the  evaluation  strategy  should  be  included  in  the  integrated  schedule  to  ensure  appropriate 
visibility  and  the  allocation  of  adequate  time  and  resources.  The  details  of  developing  or 
modifying  the  models  and  simulations  need  not  be  addressed  in  the  TEMP.  However, 
site  installation  schedules  and  configuration  releases  should  be  depicted  on  the  Integrated 
Test  Program  Schedule  and  addressed  in  Part  n  of  the  TEMP.  This  will  indicate  the 
STEP  resources  that  will  be  provided  and  when  they  will  be  available  to  support  the 
evaluation  strategy. 


Part  III  Developmental  Test  and  Evaluation  Outline 

Part  in  addresses  the  evaluation  strategy,  which  examines  the  degree  to  which  the 
system  meets  requirements  as  both  the  system  and  the  requirements  mature.  The  majority 
of  the  iterative  work  of  STEP  is  accomplished  during  the  DT  program.  It  is  here  that 
STEP  is  employed  to  incrementally  develop  and  test  the  system,  making  refinements  to 
the  system  as  well  as  the  M&S  tools.  A  key  element  of  STEP  is  the  concurrent  growth 
and  maturation  of  capabilities  in  both  the  system  under  development  and  STEP  resources. 

Early  application  of  STEP  that  helps  refine  the  system  requirements  will  also  ex¬ 
amine  the  applicability  of  existing  M&S  to  support  the  evaluation  strategy.  M&S  should 
be  used  during  the  early  phases  of  system  development  to  provide  information  on  per¬ 
formance  drivers,  CTPs,  and  technical  risk  areas  such  as  system  integration  and  system 
interfaces.  This  information  should  be  used  to  conduct  informed  cost-performance  trade 
studies. 


The  TEMP  should  also  document  how  the  early  use  of  M&S  will  aid  in  assessing 
vulnerability  and  lethality.  Documentation  relating  to  statutory  lethality  and  vulnerability 
(LFT&E)  should  be  included  in  Part  IV.  LFT&E  differs  from  OT&E  in  that  live  fire  tests 
result  in  some  degree  of  damage  to  the  target  system.  Legislation  and  DoD  5000.2-R  rec¬ 
ognize  that  both  modeling  and  component  level  testing  should  have  an  impact  on  the  pro¬ 
gram  early  enough  to  correct  design  deficiencies  before  LRIP.  Models  based  upon  the 
understanding  of  underlying  physical  phenomena  offer  the  greatest  potential  to  assess 
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vulnerability  and  lethality  interactions.  Fire,  explosion,  hydrodynamic  ram,  dynamic  in¬ 
stabilities,  and  hypervelocity  interactions  are  possible  candidates  for  physics-based  mod¬ 
eling. 


The  TEMP  should  address  STEP  resources  that  will  help  identify  areas  where  data 
is  needed  and  where  M&S  development  is  needed.  The  TEMP  should  be  updated  to  re¬ 
flect  the  results  of  trade  studies  that  affect  the  evaluation  strategy,  the  test  strategy,  and 
the  test  and  evaluation  resources.  The  TEMP  should  also  identify  areas  where  actual 
testing  either  can  be  augmented  by  M&S  or  used  to  validate  the  models  and  simulations. 
The  program  offices  should  use  this  information  to  help  determine  and  document  in  Part 
in  of  the  TEMP  the  appropriate  analysis  methods,  instrumentation  needs  (particularly 
with  new  technology),  and  data  sources  to  support  the  evaluation  strategy. 

Validation  of  the  models  and  simulations  and  data  certification  are  integral  parts 
of  the  evaluation  strategy  and  much  of  the  information  gathered  during  the  early  applica¬ 
tion  of  M&S  can  provide  the  required  justification  for  subsequent  use  of  M&S.  The 
TEMP  should  summarize  the  M&S  W&A  and  the  data  certification  to  be  conducted.  If 
the  W&A  and  certification  methods  or  the  number  of  M&S  assets  to  be  validated  is  ex¬ 
tensive,  this  information  may  be  contained  in  a  separate  document  with  a  summary  and  a 
reference  in  the  TEMP. 


Part  IV  Operational  Test  and  Evaluation  Outline 

Part  IV  addresses  the  evaluation  used  to  determine  if  the  system  is  operationally 
effective  and  operationally  suitable.  STEP  permits  the  evaluation  strategy  to  include  an 
examination  of  operational  issues  early  in  development.  Using  M&S,  analysts  should 
examine  system  performance  from  previously  unavailable  perspectives.  Using  M&S 
analysts  can  control  introduction  of  unique  variables  to  examine  “what  if’  performance 
issues. 


The  TEMP  should  reflect  the  results  of  these  initial  operational  insights  and  the 
invaluable  information  on  the  data  collection  requirements,  performance  requirements, 
thresholds,  and  objectives  as  well  as  on  the  appropriateness  of  evaluation  measures  and 
criteria.  As  the  system  and  STEP  resources  mature,  such  information  increases  confi¬ 
dence  that  the  system  can  be  certified  ready  for  lOT&E.  The  TEMP  should  reflect  the 
integration  of  models,  simulations,  and  test  events  to  obtain  the  most  credible  data  with 
which  to  conduct  a  comprehensive  evaluation  of  performance. 

The  TEMP  documents  an  evaluation  plan  that  carefully  considers  the  selection  of 
appropriate  evaluation  measures  and  the  ability  to  obtain  and  compare  credible  data  col¬ 
lected  from  both  simulations  and  test  events.  The  TEMP  should  include  a  discussion  of 
the  mission-level  and  engagement-level  models  and  simulations,  that  will  be  used  to 
identify  COIs  and  the  operationally  significant  MOEs  and  MOPs,  so  CTPs  can  be  de¬ 
rived. 
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STEP  facilitates  defining  the  COIs  and  conducting  OAs  of  system  capabilities 
throughout  system  development.  The  TEMP  should  document  how  these  OAs  will  be 
supported  with  the  iterative  application  and  maturation  of  models  and  simulations.  This 
process  will  provide  the  confidence  and  the  data  to  develop  operational  concepts  early,  to 
evaluate  software  functional  maturity,  to  evaluate  component,  sub  system  and  system  in¬ 
terface  performance,  to  evaluate  human  factors,  and  to  certify  system  readiness  for  dedi¬ 
cated  lOT&E.  The  TEMP  should  also  document  how  the  early  use  of  M&S  will  aid  in 
assessing  the  operational  impact  of  suitability  issues  (the  "ilities")  and  logistics,  and  in 
determining  initial  tactics,  training,  and  procedures. 

Part  IV  of  the  TEMP  should  document  the  integrated  use  of  accredited  models  and 
simulations  with  OT  to  increase  the  knowledge  and  understanding  of  the  capabilities  and 
the  limitations  of  the  system  as  it  will  be  employed.  Knowing  the  limitations  of  a  system 
from  models,  simulation,  and  tests  is  important  to  the  effective  employment  of  the  sys¬ 
tem.  The  final  tactics,  training,  and  procedures  for  employment  will  ultimately  be  based 
upon  a  thorough  understanding  of  the  system  capabilities  and  limitations  determined  from 
the  comprehensive  and  integrated  use  of  models,  simulations,  and  operational  tests. 


Part  V  Test  and  Evaluation  Resource  Summary 

Part  V  addresses  the  resources  needed  to  ensure  the  evaluation  strategy  can  be  car¬ 
ried  out  effectively,  and  is  adequate  to  determine  the  system  under  development  is  opera¬ 
tionally  effective,  suitable,  and  survivable.  STEP  requires  greater  emphasis  in  identify¬ 
ing,  in  the  TEMP,  the  models  and  simulations,  and  the  associated  W&A  information 
that  will  be  integrated  with  real  test  events  to  obtain  credible  data  for  evaluating  perform¬ 
ance.  Step  resources  that  should  be  integrated  into  Part  V  of  the  TEMP  include: 

•  Facilities  (government,  industrial  and  academic), 

•  Threat  representations  including  simulations, 

•  Specific  existing  models  and  simulations, 

•  Specific  models  and  simulations  to  be  developed,  and 

•  Resources  to  conduct  W&A,  including  instrumentation  needs. 

STEP  alters  the  focus  of  the  types  of  resources  required  to  develop  and  carry  out  a 
comprehensive  evaluation  strategy.  Early  and  iterative  use  of  M&S  during  development 
can  identify  critical  test  issues  and  focus  the  expenditure  of  test  assets  on  critical  test 
events.  M&S  may  be  used  to  increase  the  efficiency  of  test  events  by  conducting  “dry 
runs”  that  can  identify  procedural,  data  collection,  and  analytical  difficulties  before  ex¬ 
pending  valuable  test  assets.  This  approach  and  the  associated  resources  should  be 
documented  in  the  TEMP. 
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The  STEP  resources  used  in  each  phase  of  acquisition  may  differ  as  the  program 
matures.  The  STEP  resources,  the  data  generated  by  simulations,  and  the  data  (from  real 
test  events)  used  to  validate  the  M&S  should  be  archived  for  future  use.  Such  use  may 
occur  later  in  the  life  cycle  of  the  program  or  may  be  used  by  other  programs.  The  re¬ 
sources  required  to  W&A  the  models  and  simulations;  the  resources  required  to  obtain, 
maintain^  and  M&S;  and  the  resources  required  to  archive  data  for  the  M&S  should  be 
included  in  Part  V  of  the  TEMP. 


Annex  A — Bibliography 

The  TEMP  should  not  duplicate  information  contained  in  other  sources  (e.g., 
simulation  support  documents  or  VV&A  documents).  The  TEMP  should  summarize 
such  information  and  refer  to  the  source  documents.  Verification  documents,  validation 
documents,  and  accreditation  documents  for  each  model  and  simulation  used  in  STEP 
should  be  included  in  the  bibliography. 
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CHAPTER  4 


STEP  Tools,  Standards,  and  Resources 


4.1  Introduction 


Many  of  the  tools  needed  for  STEP  already  support  the  systems  acquisition  proc¬ 
ess,  both  in  DoD  and  industry.  Often  these  tools  have  been  developed  for  a  specific 
function  or  purpose  with  little  thought  to  their  integration  with  the  entire  acquisition 
process.  As  dramatic  advances  in  supporting  technologies  make  these  tools  more  power¬ 
ful  and  less  expensive,  and  as  declining  resources  and  changing  priorities  make  it  essen¬ 
tial  to  find  better  ways  to  develop  and  field  new  systems,  the  use  of  these  tools,  and  of 
improved  processes  that  exploit  their  contribution,  is  expanding  rapidly. 

4.2  Tools 


The  descriptions  of  STEP  tools  that  follow  are  not  all-inclusive,  but  are  provided 
to  show  the  range  of  types  that  exist  and  illustrate  potential  applications  to  STEP.  What 
is  desired  is  the  effective  integration  of  productive  STEP  tools  and  the  reuse  of  many  of 
these  tools  throughout  the  acquisition  process.  These  can  be  viewed  in  the  context  of  the 
activities  detailed  in  the  acquisition  strategy,  some  examples  of  which  are  noted  in  Figure 
2-1.  When  planning  the  evaluation  strategy,  it  is  advantageous  to  begin  planning  for 
funding  and  consider  integration  across  the  functional  areas  as  early  in  the  process  as  pos¬ 
sible. 


Models  and  Simulations 

Models  and  simulations  underlie  all  other  STEP  tools.  M&S  supports  early  in¬ 
volvement  of  the  tester  and  evaluator  in  the  system  concept  development  stage  to  help 
formulate  functional  requirements  in  a  testable  manner.  STEP  resources  in  an  evaluation 
strategy  include  authoritative  and  validated  representations  of  the  threat,  e.g.,  both  radio 
frequency  (RF)  and  infrared  (ER)  missiles,  threat  radars,  aircraft,  and  the  associated  com¬ 
mand  and  control.  Other  important  resources  are  models  or  simulations  of  the  environ¬ 
ment,  including  terrain,  atmosphere,  space,  and  ocean.  These  and  other  simulation  tools 
are  used  to  plan,  rehearse,  extend  and  evaluate  live  testing  activities. 

M&S  includes  a  wide  range  of  computationally  based  activities  such  as  mathe¬ 
matical  models  exercised  on  digital  computers.  As  discussed  in  Figure  4-1,  models  can 
be  viewed  in  a  hierarchy  from  engineering  and  engagement  level  models  to  mission  and 
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campaign  level  models.  When  planning  the  evaluation  strategy,  the  program  office 
should  consider  which  types  of  models  fit  the  needs  for  the  developing  system. 


Figure  4-1  illustrates  the  different  levels  of  models  and  simulations  required,  and 
depicts  how  they  form  a  hierarchy. 

•  Engineering-level  models  often  involve  the  use  of  three-dimensional  CAD/CAM/ 

or  computational  fluid  dynamics  or  other  computational  approaches  to  consider 
such  effects  as  signatures,  extreme  environments,  mobility,  and  fatigue. 

•  Engagement-level  models  are  used  to  explore  such  issues  as  end-game  lethality,  fire¬ 
power,  manufacturing,  and  producibility. 

•  Mission/battle-level,  system-on-system  models  support  analyses  of  system  perform¬ 
ance,  platform  engagement,  survivability,  and  mobility. 

•  Theater/campaign-level,  force-on- force  models  provide  insights  into  a  system’s 
contribution  to  force  effectiveness,  as  well  as  force  structure  and  logistics  require¬ 
ments. 

M&S  also  includes  physical  representations  as  system  models,  and  large  com¬ 
puter-based  simulations  that  predict  system  performance.  The  development  of  the  models 
that  describe  a  system,  and  the  M&S  that  predict  its  performance,  should  be  driven  by  the 
identification  and  refinement  of  system  requirements.  For  example,  the  MNS  for  a  major 
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weapon  system  initiates  the  selection  or  development  of  models  and  simulations  to  assess 
the  performance  of  the  proposed  system  in  theater/campaign,  mission/battle,  and  engage¬ 
ment-level  conflicts.  The  top-level  system  requirements  of  automated  information  sys¬ 
tems  (AIS)  drive  tiie  development  of  top-level  models  of  computer  hardware  and  soft¬ 
ware,  data,  and  telecommunication  systems  that  can  be  used  in  simulations  to  predict 
functions  such  as  collecting,  processing,  transmitting,  and  displaying  information. 

M&S  can  be  used  to  design  a  better  test  program,  add  realism  to  test  scenarios, 
extrapolate  results  of  live  testing,  and  understand  aspects  of  system  performance  observed 
during  live  testing.  This  can  reduce  time,  resources,  and  risk  to  an  acquisition  program. 
In  some  cases,  M&S  is  the  only  way  to  conduct  system  assessment  and  is  the  only  way  to 
generate  “reproducible”  scenarios  and  conditions.  The  major  limitation  associated  with 
models  and  simulations  is  credibility,  which  is  addressed  in  more  detail  in  Section  4.5.  A 
technical  challenge  is  linking  or  aggregating  results  from  engineering  models  into  higher 
level  models.  Additionally,  many  existing  models  and  computer  simulations  are  based  on 
empirical  data  rather  than  physical  principles.  M&S  output  may  be  more  readily  accepted 
if  M&S  tools  are  based  on  proven  principles  of  physics,  validated  by  experimental  data. 

Advanced  Distributed  Simulation  (ADS).  ADS  is  an  environment  in  which 
simulations  are  linked  to  produce  large  synthetic  environments  within  which  large  num¬ 
bers  of  subjects  can  interact  in  real  time.  The  principal  characteristics  of  ADS  are  that 
participating  simulations  are  physically  separated,  are  linked  electronically,  and  share  a 
common  view  of  their  electronic  environment.  The  responses  elicited  from  each  simula¬ 
tion  are  seen,  interpreted,  and  acted  upon  by  the  other  simulations  in  near  real  time. 

ADS  offers  the  potential  to  link  multiple  non-collocated  T&E  for  an  evaluation. 
This  could  include  multiple  systems  being  linked  and  operating  within  a  realistic  syn¬ 
thetic  environment,  thus  creating  a  virtual  system.  The  benefits  of  this  linkage  are  early 
interoperability  and  compatibility  evaluation,  and  early  user  input  to  the  system  operating 
in  concert  with  other  new  systems  in  a  synthetic  environment.  Cost  savings  could  be  re¬ 
alized  through  early  testing,  reduced  need  for  transportation  and  repositioning,  and  re¬ 
duced  acquisition  cycle  time.  Limitations  of  ADS  include  latency,  or  the  communication 
delay  between  participants,  and  the  immaturity  of  the  technology.  The  challenges  of 
W&A  are  compounded  by  the  distributed  nature  of  the  ADS  network;  the  linked  net¬ 
work  will  require  W&A  in  addition  to  each  stand-alone  M&S  participant. 

Computational  Fluid  Dynamics  (CFD).  CFD  is  a  numerical  approach  for 
modeling  the  dynamics  of  a  fluid  flow  in  and  around  solid  objects.  In  aircraft  applica¬ 
tions,  it  can  be  applied  to  model  stores  separation  to  analyze  aircraft-store  loading,  safe 
carriage  and  separation,  safe  escape  and  ballistic  accuracy.  The  cost  associated  with  certi¬ 
fication  of  stores  for  release  from  an  aircraft  is  very  high.  CFD  is  often  used  as  a  tool  in 
direct  support  of  wind  tunnel  and  flight  testing.  Integrating  the  modeling  tools  directly 
with  ground  and  flight  tests  enables  the  tester  to  design  a  better  test  program,  validate 
and/or  extrapolate  the  results,  and  assist  in  decision  making  for  a  more  efficient  or  effec¬ 
tive  test.  CFD  can  be  applied  to  investigate  anomalies  observed  during  wind  tunnel  test¬ 
ing  at  significantly  reduced  cost  over  repeat  wind  tunnel  trials.  The  principal  limitation  of 
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CFD  is  credibility.  Optimal  application  of  CFD  is  in  conjunction  with  wind  tunnel  and 
flight  test/analysis  effort  employing  the  model-test-model  approach. 

Simulators.  The  word  “simulator”  has  a  number  of  meanings: 

(1)  A  family  of  equipment  used  to  represent  threat  systems  in  testing  and 
training.  A  threat  simulator  has  one  or  more  characteristics  which,  when  de¬ 
tected  by  human  senses  or  man-made  sensors,  provide  the  appearance  of  an  ac¬ 
tual  threat  system  within  a  known  degree  of  realism. 

Threat  simulators  are  particularly  useful  to  testers  in  creating  realistic  environ¬ 
ments  and  as  a  means  to  mitigate  limitations  to  the  scope  of  testing  because  actual  threat 
systems  operated  by  trained  “enemy”  personnel  may  not  be  available  for  realistic  testing. 
Threat  simulators  can  be  emissions,  signatures  or  radar  returns  synthetically  injected  into 
a  controlled  laboratory  environment,  or  they  could  be  friendly  units  playing  the  role  of 
threat  forces  in  doctrine  or  performance.  In  some  cases,  threat  hardware  is  used.  M&S 
“simulators”  may  produce  their  greatest  utility  in  providing  the  electronic  means  to  gen¬ 
erate  sufficient  numbers  of  threat  forces  to  provide  a  meaningful  evaluation  of  new  sys¬ 
tem  capability. 

(2)  A  human-in-the-loop  device  that  provides  the  conditions  and  environment 
of  a  system  to  accurately  produce  aspects  of  system  performance  and  operation  to 
estimate  human  performance  associated  with  the  system  and  to  conduct  training 
and  develop  tactics,  techniques,  and  procedures. 

Simulators  in  this  second  meaning  are  commonly  employed  as  training  devices, 
but  have  numerous  applications  in  support  of  testing,  system  operation,  and  tactics. 
Simulators  developed  concurrently  with  the  system  provide  the  means  to  obtain  user 
feedback  on  operational  and  human  factors  issues.  When  integrated  with  other  systems 
via  ADS,  great  insight  into  the  operational  aspects  of  the  system  can  be  obtained. 

(3)  A  physical  representation  of  a  system  which  can  demonstrate  certain  as¬ 
pects  of  system  operation. 

An  example  of  this  third  type  of  simulator  is  the  Army’s  Firing  Impulse  Simula¬ 
tor.  This  device  mechanically  duplicates  the  recoil  shock  of  firing  large  caliber  weapons 
without  actual  firing.  Environment  effects  (e.g.,  blast  overpressure  and  noise)  and  am¬ 
munition  cost  are  avoided  while  representative  dynamic  firing  impulses  are  produced. 

Stimulators.  A  stimulator  is  a  simulation  used  to  provide  an  external  stimulus  to 
a  system  or  subsystem.  The  output  of  the  simulation  is  used  to  “stimulate”  the  system 
(hardware  and/or  software)  being  evaluated  for  purposes  of  analysis.  The  unit  under  test 
may  be  in  a  HWIL  configuration,  test  stand,  or  live  field  conditions.  Limitations  include 
the  degree  of  realism  of  the  stimulator,  lack  of  tactics  applied,  and  total  threat  system  rep¬ 
resentation  in  fit,  form  and  function  to  replicate  the  threat. 

Measurement  Facilities 
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Measurement  facilities  are  used  to  quantify  or  measure  parameters  (such  as  thrust, 
radar  cross  section  (RCS),  and  drag)  of  a  test  article  in  precise  terms.  Types  of  such  fa¬ 
cilities  are  wind  tunnels,  anechoic  chambers,  captive  carry  systems,  RCS  facilities,  anten¬ 
nae  pattern  ranges,  and  engine  thrust  stands.  Test  articles  may  he  scale  models,  mockups, 
or  actual  hardware.  Examples  of  measurement  facilities  are  wind  tunnels,  anechoic 
chambers,  and  captive  carry  facilities. 

Wind  Tunnel.  A  wind  tunnel  is  a  chamber  through  which  air  is  forced  at  con¬ 
trolled  speeds  so  that  its  effect  on  an  object  can  be  studied.  A  wind  tunnel  can  be  used  at 
different  times  in  a  system’s  development  to  analyze  air  stream  effects.  This  testing  is 
especially  relevant  for  flight  safety  evaluation  and  issue  resolution.  This  evaluation  can 
be  augmented  with  computational  fluid  dynamics  (CFD),  flight  testing,  and  captive  carry 
evaluation.  Limitation  of  wind  tunnel  use  are  the  cost  and  availability  of  the  facility  and 
limitations  on  the  size  of  the  weapon  that  can  be  accommodated.  Due  to  the  cost  of  de¬ 
velopment  and  operation,  most  wind  tunnels  cannot  accommodate  full-scale  systems. 
This  forces  the  scaling  down  of  systems  to  less  than  full-scale  models  for  testing  and  im¬ 
poses  limitations  on  conclusions  drawn  from  wind  tunnel  tests. 

Anechoic  Chamber.  An  anechoic  chamber  is  a  facility  that  provides  an  essen¬ 
tially  echo-free  environment  at  various  electromagnetic  frequencies  for  laboratory  meas¬ 
urements  including  RCS,  antenna  pattern,  and  passive  radar  augmentation.  Anechoic 
chambers  provide  a  “pure”  environment  for  testing,  and  the  additional  benefit  of  secure 
testing  with  no  external  emissions  which  can  be  monitored  by  unauthorized  personnel. 
Weapons  system  size  limitations  may  preclude  some  types  of  testing. 

Captive  Carry.  Captive  carry  of  stores  or  vehicles  is  a  nondestructive  means  of 
obtaining  data  on  air-capable  vehicles.  Instead  of  free-flying  the  vehicle,  it  is  mounted  on 
a  parent  aircraft.  This  “hosf’  aircraft  provides  the  motion,  power,  and  support  services 
required  to  operate  the  “guest.”  Additionally,  the  host  aircraft  often  collects  the  data  ob¬ 
tained  for  later  analysis.  A  captive  carry  store  or  vehicle  is  normally  not  damaged  and 
can  be  examined,  analyzed,  evaluated,  and  reused. 


Hardware/Software  in  the  Loop 

Hardware/Software  In  the  Loop  (HWIL/SWIL)  is  a  hybrid  simulation  that  in¬ 
cludes  actual  system  (prototype  or  production)  hardware  or  software  in  conjunction  with 
digital  models  and  external  stimuli  to  demonstrate  the  operation  and  function  of  the 
hardware/software  within  an  environment  simulating  actual  operating  conditions. 
HWIL/SWIL  can  be  used  to  demonstrate  new  technology;  evaluate  designs,  concepts,  and 
prototypes;  and  show  the  integration  of  hardware  and  software.  It  allows  early  evaluation 
without  the  expenditure  of  live  test  resources,  facilitates  live  test  development  by  identi¬ 
fying  critical  test  conditions,  and  can  be  used  for  development  of  data  collection  plans. 
The  laboratory  environment  may  provide  for  easier  data  collection  as  a  result  of  better 
access  to  components. 


4-5 


SWIL  may  provide  the  only  method  to  examine  complex  software  and  the  algo¬ 
rithms  and  logic  flow  of  programming  in  adequate  detail.  If  the  operational  conditions  or 
environment  of  live  testing  are  such  that  conditions  necessary  to  manifest  a  fault  do  not 
occur,  the  fault  may  not  be  detected.  If  undetected,  the  fault  can  be  duplicated  or  promul¬ 
gated  across  other  interfacing  systems.  Correction  of  faults  after  software  release  and  in¬ 
tegration  in  deployed  systems  is  difficult  and  expensive.  SWIL  is  the  best  opportunity  for 
fault  detection  and  software  risk  reduction  prior  to  release,  and  is  one  of  the  most  cost- 
effective  methods  of  fault  correction. 

For  early  phases  of  operational  testing,  HWIL  can  demonstrate  potential  effec¬ 
tiveness  and  suitability  for  designs  that  exist  in  a  pre-production  or  breadboard  stage  of 
development.  User  feedback  on  HWIL/SWIL  performance  can  be  used  to  improve  the 
design  early  in  the  process.  The  reduction  in  risk  by  early  system  integration  can  be  of 
significant  value  to  the  program  manager.  In  some  programs,  complete  HWIL  simula¬ 
tions  housed  in  elaborate  system  integration  laboratories  have  provided  fully  integrated 
system  testing  prior  to  platform  installation  and  offer  the  additional  benefit  of  crew  per¬ 
formance  evaluation.  The  principal  limitation  to  HWIL/SWIL  simulation  is  realism  in 
the  operating  environment  and  credibility  of  the  system  representation.  Use  of  ADS  to 
bring  realistic  synthetic  environments  into  the  testing  loop  and  robust  W&A  of  the  sys¬ 
tem  representation  can  mitigate  these  problems. 


System/Software  Integration  Laboratory 

A  System/Software  Integration  Laboratory  (SIL)  is  a  facility  that  supports  the  in¬ 
tegration  of  system  components  and/or  software  in  a  laboratory  environment  for  devel¬ 
opment,  experiments,  and  testing.  The  integration  laboratory  “simulates”  (or  replicates)  a 
system  to  a  known  extent  and  allows  the  modification/addition  of  component  hard¬ 
ware/software  for  use  without  many  of  the  restrictions  or  difficulties  that  would  be  en¬ 
countered  using  actual  system  hardware  or  host  platforms.  The  SIL  is  the  physical  sup¬ 
port  structure  and  components  that  make  HWIL/SWEL  testing  and  evaluation  possible. 

SILs  can  be  used  by  prime  contractors  to  evaluate  the  compatibility  and  interoper¬ 
ability  of  subsystems  and  components  developed  by  various  subcontractors.  A  variety  of 
computer  simulations,  real-world  equipment,  and  images  generate  scenarios  and  envi¬ 
ronments  to  test  component  interaction  while  performing  mission  tasks. 


Installed  System  Test  Facilities 

An  Installed  System  Test  Facility  (ISTF)  is  a  facility  where  entire  systems  or  sub¬ 
systems  get  their  first  workout  in  the  environment  in  which  they  will  operate  (e.g.,  inside 
an  aircraft).  A  full  capability  ISTF  has  the  ability  to  mix  a  complete  spectrum  of  players 
from  synthetic  (digital  models)  to  real  (actual  hardware)  to  hybrid  (a  combination  of 
both).  It  has  the  ability  to  provide  multilevel  threat  simulations  (open-loop  and  closed- 
loop  signal  simulators,  including  actual  or  simulated  threat  system  hardware).  It  also  has 
the  ability  to  provide  simulations  of  all  elements  a  system  would  be  expected  to  oper- 
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ate  in  the  real  world.  The  Navy’s  Air  Combat  Environment  Test  and  Evaluation  Facility 
(ACETEF)  at  Patuxent  River,  Maryland,  is  an  example  of  an  ISTF. 


Live  Test  Ranges 

Instrumented  test  ranges  permit  system-level  tests  in  a  real-world,  dynamic  envi¬ 
ronment.  The  Major  Range  and  Test  Facility  Base  (MRTFB)  is  a  set  of  test  installations, 
facilities,  and  ranges  which  are  regarded  as  “national  assets.”  MRTFB  assets  are  sized, 
operated,  and  maintained  primarily  for  DoD  T&E  missions.  A  variety  of  ranges  are 
available  which  can  facilitate  fully  operational,  system-level  testing.  They  are  instru¬ 
mented  for  data  collection,  time-space-position  information,  positive  control,  and  safety. 
Typically,  both  real  or  simulated  targets  and  interactive  threats  are  available.  The  ranges 
employ  actual  and  simulated  systems  to  achieve  high  levels  of  credibility  approaching 
representation  of  combat  operations.  Examples  of  live  test  ranges  are  listed  below: 

•  Air  Force:  Utah  Test  and  Training  Range,  Nellis  AFB  Range  Complex,  and  Arma¬ 
ment  Systems  Test  Environment  and  Gulf  Range 

•  Navy:  Atlantic  Underwater  Test  &  Evaluation  Center,  and  Naval  Air  Warfare  Center 
Weapons  Division-China  Lake 

•  Army:  Kwajalein  Missile  Range,  White  Sands  Missile  Range,  and  Electronic  Proving 
Ground. 


4,3  High  Performance  Computing 


High  performance  computing  (HPC)  is  a  key  enabling  technology  whose  use  con¬ 
tinues  to  increase  rapidly  throughout  the  DoD.  High  performance  computing,  as  defined 
here,  refers  to  computer  system  capabilities,  typically  in  terms  of  speed  and  size,  that  are 
within  two  or  three  orders  of  magnitude  of  the  very  best  computers  available.  The  in¬ 
creasing  fidelity  of  evolving  modeling  and  simulation,  including  interacting  physical  and 
engineering  principles,  requires  HPC  capabilities  to  estimate  performance  and  effective¬ 
ness,  especially  in  areas  where  environmental,  legal,  safety,  or  technical  restrictions  affect 
actual  testing. 


High  performance  computing  capabilities  are  needed  in  three  major  T&E  areas: 

1.  System  Modeling  and  Simulation  -  to  execute  complex,  high-fidelity  models 
and  simulations  representing  the  system  under  test,  the  natural  environment, 
and  the  threat  environment. 
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2.  High  Performance  Range  Instrumentation— to  allow  range  instrumentation 
systems  to  perform  “intelligent’  data  processing,  such  as  machine  vision  tech¬ 
niques,  automating  and  refining  data  collection  and  analysis. 

3.  High  Performance  Databases— to  manage,  process,  and  analyze  large  data 
sets. 

Information  on  the  availability  of  HPC  resources,  their  capabilities  and  employ¬ 
ment,  can  be  found  in  the  DoD  Test  and  Evaluation  High  Performance  Computing  Mod¬ 
ernization  Plan. 


4.4  Standards 

Interoperability  of  STEP  tools  is  essential  to  assure  proper  evaluation  of  subsys¬ 
tems,  systems,  and  systems  of  systems.  A  proliferation  of  M&S  that  cannot  exchange 
data,  or  that  require  extensive  modifications  to  interoperate,  is  counterproductive.  Initia¬ 
tives  are  in  process  to  facilitate  interoperability.  Standards  are  also  useful  in  facilitating 
reuse  of  STEP  tools  across  weapons  systems  programs. 


Common  Technical  Framework 

A  Common  Technical  Framework  (CTF)  is  the  first  objective  cited  in  the  DOD 
M&S  Master  Plan.  Its  purpose  is  to  facilitate  reuse  and  interoperability  of  M&S  by  pro¬ 
viding  some  pillars  of  commonality;  an  architecture  to  which  M&S  must  conform,  a  ba¬ 
sis  for  the  development  of  consistent  and  authoritative  simulation  representations,  and 
data  standards  to  provide  common  representation  of  data  across  M&S.  Principal  compo¬ 
nents  of  the  CTF  are  a  High  Level  Architecture,  a  Conceptual  Model  of  the  Mission 
Space,  and  Common  Database  Standards. 

High  Level  Architecture.  The  High  Level  Architecture  (HLA)  has  been  desig¬ 
nated  the  standard  architecture  for  M&S  throughout  DoD.  HLA  provides  a  common  ar¬ 
chitecture  for  reuse  of  simulations.  It  is  based  on  the  premise  that  no  single  model  or 
simulation  can  satisfy  all  uses  and  users  in  DoD  at  all  levels  of  resolution.  An  individual 
simulation  or  set  of  simulations  developed  for  one  purpose  can  be  applied  to  another  ap¬ 
plication  under  the  HLA  federation  concept  that  calls  for  a  selectable  set  of  interacting 
simulations.  The  intent  of  the  HLA  is  to  provide  a  structure  that  will  support  reuse  of  ca¬ 
pabilities  available  in  different  simulations,  ultimately  reducing  the  cost  and  time  required 
to  create  a  synthetic  operating  environment  for  a  new  purpose. 

Common  Model  of  the  Mission  Space.  The  Common  Model  of  the  Mission 
Space  (CMMS)  is  a  simulation-neutral  view  of  the  real  world,  and  acts  as  a  bridging 
function  between  the  warfighter,  who  owns  die  combat  process  and  serves  as  the 
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authoritative  source  for  validating  CMMS  content,  and  simulation  developers.  Addition¬ 
ally,  the  CMMS  provides  a  common  viewpoint  and  serves  as  a  vehicle  for  communica¬ 
tions  among  warfighters,  doctrine  developers,  trainers,  C4I  developers,  analysts,  and 
simulation  developers.  Such  a  foundation  allows  all  concerned  parties  to  be  confident 
that  DoD  simulations  are  founded  in  operational  realism.  The  CMMS  will  require  reli¬ 
ance  on  authoritative  sources  and  serve  as  the  means  for  capturing,  sharing,  and  evolving 
information. 

Common  Database  Standards.  The  overarching  objective  is  to  enable  data 
suppliers  to  provide  the  community  affordable,  timely,  verified,  and  validated  data  to 
promote  reuse  and  sharing  of  data,  interoperability  of  models  and  simulations,  and  im¬ 
proved  credibility  of  M&S  results.  The  policies,  procedures,  and  methodologies  for  data 
standards  form  general  guidance  for  data  used  in  environmental,  systems  and  human  be¬ 
havior  representations. 

Joint  Technical  Architecture 

The  DoD  Joint  Technical  Architecture  (JTA)  is  a  key  piece  of  the  Department's 
overall  strategy  to  achieve  interoperability.  The  JTA  specifies  a  set  of  performance- 
based,  primarily  commercial,  information  processing,  transfer,  content,  format  and  secu¬ 
rity  standards.  These  standards  specify  the  logical  interfaces  in  command,  control,  and 
intelligence  systems  and  the  communications  and  computers  (C4I)  that  directly  support 
them.  The  JTA  is  a  practical  document,  identifying  standards  where  products  are  avail¬ 
able  today.  JTA  (Version  1.0)  is  mandatory  for  all  emerging  systems  and  system  up¬ 
grades.  HLA  is  being  included  along  with  other  M&S  standards  in  the  JTA. 


Joint  Modeling  and  Simulation  System 

The  Joint  Modeling  And  Simulation  System  (JMASS)  is  a  program  to  develop 
and  deliver  a  distributed,  object-oriented  M&S  architecture  and  system  focused  on  the 
tactical  level  of  war  (mission-  and  engagement-level  simulation).  It  is  designed  to  pro¬ 
vide  a  flexible,  standardized  M&S  tool  to  support  a  wide  range  of  simulation  require¬ 
ments  throughout  the  life  cycle  of  a  weapon  system.  It  supports  concept  design,  trade 
studies,  performance  evaluations,  and  tactics  analysis  and  development. 

4.5  Support  Activities 

M&S  Operational  Support  Activity 

The  M&S  Operational  Support  Activity  (MSOSA)  is  chartered  to  supply  M&S  re¬ 
source  information  to  DoD  customers.  It  catalogues  M&S  resources,  databases,  exercises, 
capabilities,  and  points  of  contact  and  makes  this  information  available  to  the  DoD  com¬ 
munity.  MSOSA  resources  link  to  each  of  the  Services,  the  Information  Analysis  Centers, 
and  other  specific  M&S  sources.  Through  an  on-line  database,  information  is  readily 
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available.  MSOSA  also  provides  M&S  expertise  and  direct  access  to  the  DoD  M&S 
“corporate  knowledge  base.” 

MSOSA  users  may  be  able  to  benefit  from  the  reuse  of  existing  M&S  products 
and  databases;  obtain  easy  access  to  information  on  M&S  policies,  activities,  and  initia¬ 
tives;  coordinate  with  ongoing  M&S  “events”;  and  experience  cost  savings  through  asset 
sharing.  MSOSA  builds  databases  based  upon  the  needs  of  the  community,  and  expands 
its  capability  based  on  contacts  made  and  requests  for  services.  MSOSA  provides  a 
staffed,  help  desk  and  24-hour,  on-line  service  via  the  Internet. 


Modeling  and  Simulation  Resource  Repository 

The  Modeling  and  Simulation  Resource  Repository  (MSRR)  Web  page  provides 
several  M&S  informational  services  such  as  the  M&S  Digital  Library,  the  M&S  Exercise 
and  Event  Calendar,  and  an  M&S  search  engine  called  the  Intelligent  Mission  Support 
System.  A  linked  system  of  resources  (servers,  classified  as  nodes)  allow  connectivity, 
reuse,  and  sharing  of  M&S  resources  to  support  communication  and  information  ex¬ 
change.  A  prototype  MSRR,  leveraging  the  existing  technologies  of  internetworking  is 
established  to  provide  a  number  of  useful  services,  including  software  tools,  electronic 
documents,  government  off-the-shelf  applications,  a  repository  of  M&S  data  models,  and 
directories  and  catalogs. 


M&S  Executive  Agents 

M&S  Executive  Agents  (MSEA)  are  DoD  Components  or  Agencies  who  have 
been  assigned  management  responsibility  by  the  Under  Secretary  of  Defense  (Acquisition 
and  Technology)  for  the  development  and  maintenance  of  a  specific  area  of  M&S  appli¬ 
cation,  including  relevant  standards  and  data  bases,  used  by  or  common  to  many  models 
and  simulations.  The  current  MSEA  are  shown  in  Table  4-1 . 


Table  4-1.  M&S  Executive  Agents 


Area 

Executive  Agent 

Air  and  Space 

Air  Force  AFCCC 

Ocean 

NRL  Code  7306 

Terrain 

Defense  Mapping  Agency, 

Terrain  Modeling  Project  Office 

Threat/Intelligence 

Defense  Intelligence  Agency 
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CHAPTER  5 


Summary 


The  DoD  is  placing  emphasis  on  the  use  of  Integrated  Product  and  Process  Devel¬ 
opment  (IPPD)  and  Integrated  Product  Teams  (IPTs)  as  it  develops  better  ways  to  deter¬ 
mine  what  to  buy  and  better  methods  of  buying  what  it  needs. 

Simulation  based  acquisition  (SBA)  promotes  the  robust,  collaborative  use  of 
simulation  technology  integrated  across  acquisition  phases  and  programs.  It  is  envisioned 
to  substantially  reduce  the  time,  resources,  and  risk  associated  with  the  acquisition  proc¬ 
ess  and  to  increase  the  quality,  military  utility,  and  supportability  of  systems  developed 
and  fielded.  SBA  may  involve  an  initial  investment  but  is  expected  to  save  resources 
over  the  lifecycle  of  the  system. 

The  Simulation,  Test,  and  Evaluation  Process  is  a  test  and  evaluation  answer  to 
the  DoD  challenges  of  implementing  IPPD  and  SBA.  STEP,  as  the  participation  of  T&E 
in  SBA,  focuses  on  reusability  and  interoperability,  enabling  the  acquisition  community 
to  take  a  measured  approach  in  realizing  the  objectives  of  SBA. 

Modeling  and  simulation  used  collaboratively  can  reduce  the  lifecycle  costs  asso¬ 
ciated  with  the  integration  of  complex  systems,  by  helping  the  members  of  an  IPT  iden¬ 
tify  a  program’s  technical  and  operational  challenges  as  early  as  possible  and  plan  for 
resolution  of  the  issues.  Commercial  computer  hardware  and  software,  virtual  proto¬ 
types,  and  simulations  can  aid  in  developing  a  shared  vision  of  the  proposed  system  and 
provide  a  means  for  understanding  the  complex  interactions  among  the  configuration 
items  in  the  system  design.  In  other  words,  modeling  and  simulation  can  be  a  key  con¬ 
tributor  to  the  Integrated  Product  and  Process  Development  (IPPD)  that  the  Secretary  of 
Defense  has  directed. 

Substantial  evidence  has  already  accumulated  as  to  the  value  of  modeling  and 
simulation  in  acquisition.  Both  commercial  and  military  system  development  programs 
provide  pervasive  evidence  of  tangible  results  that  can  be  measured  in  improvements  in 
cost,  schedule,  productivity,  and  quality/performance.  STEP  is  not  merely  an  incremental 
step  beyond  current  system  engineering  methods  and  tools — ^it  represents  a  major  shift 
toward  a  comprehensive,  integrated  environment  of  M&S  within  the  acquisition  process. 

Key  in  STEP  is  the  integrated  simulation,  engineering  management  and  test  fo¬ 
cused  on  program  area  success.  The  product  from  STEP  then  becomes  information  and 
insight  that  can  be  shared  concurrently  across  functional  areas  and  the  acquisition  com¬ 
munity  in  the  spirit  of  IPPD. 

STEP  enhances  the  acquisition  process,  fully  integrating  the  use  of  models,  simu¬ 
lations,  and  test  events,  and  implements  the  intent  of  DoD  5000.2.  It  provides  early  and 
continuous  information  on  the  joint  military  worth  of  a  system  so  that  the  total  life  cycle 
is  more  effective  and  efficient.  STEP  begins  by  focusing  on  the  mission  level,  collecting 


5-1 


models  and  simulations  that  can  be  used  to  understand  how  the  new  system  might  affect 
the  mission;  identifying  and  prioritizing  critical  information  needs  for  the  operational 
concept  developers  and  engineers;  and  integrating  the  use  of  models,  simulations,  and 
tests  to  provide  that  information.  STEP  can  be  applied  throughout  the  system  life  cycle, 
and  can  provide  consistent,  comparable  performance  information  for  the  requirements, 
acquisition,  test  and  evaluation,  and  training  communities. 

In  order  for  this  process  to  be  fully  realized,  the  program  office  must  work  with 
the  T&E  community  to  develop  an  effective  evaluation  strategy  which  will  use  simulation 
and  test  resources.  Accreditation  and  certification  are  essential  and  must  be  included  in 
an  effective  evaluation  strategy.  The  program  must  resource  these  efforts  to  realize  the 
benefits  of  STEP.  To  be  successful,  it  is  imperative  that  the  program  identify  and  fund 
STEP  resources  early  while  considering  credibility  and  cost  in  order  to  best  allocate  as¬ 
sets  to  the  most  appropriate  issues.  The  wise  application  of  STEP  will  produce  objective 
and  more  comprehensive  information  regarding  performance,  suitability,  and  survivabil¬ 
ity  than  would  T«&E  alone. 

STEP  does  not  end  with  the  fielding  and  deployment  of  the  system,  but  continues 
throughout  the  operational  support  of  the  system  to  the  end  of  its  life  cycle.  STEP  results 
in  a  thoroughly  evaluated  system  with  better  understood  operational  effectiveness,  suit¬ 
ability  and  survivability.  STEP  also  results  in  a  set  of  models  and  simulations  of  known 
fidelity,  capable  of  reuse  in  other  efforts. 
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ANNEX A 


ADS 

AFCCC 

AOA 

BMD 

CE 

CM 

COI 

CTP 

DIS 

DMSO 

DMSTTIAC 

DoD 

DOmE 

DSMC 

DT 

DT&E 

EMD 

EGA 

FLIR 

FOT&E 

HITL 

HLA 

HWIL 

IOC 

lOT&E 

IPPD 

IPT 

ISTF 

ITAS 

JADS 

JDBE 

JMASS 

LFT&E 

LRIP 

LUT 

M&S 


Acronyms  and  Definitions 


Advanced  Distributed  Simulation 

Air  Force  Combat  Climatology  Center 

Analysis  Of  Alternatives 

Ballistic  Missile  Defense 

Concept  Exploration 

Configuration  Management 

Critical  Operational  Issue 

Critical  Technical  Parameter 

Distributed  Interactive  Simulations 

Defense  Modeling  and  Simulation  Office 

Defense  Modeling,  Simulation,  and  Tactical 

Technology  Information  Analysis  Center 

Department  of  Defense 

Director,  Operational  Test  and  Evaluation 

Defense  Systems  Management  College 

Developmental  Test;  Developmental  Testing 

Developmental  Test  and  Evaluation 

Engineering  and  Manufacturing  Development 

Early  Operational  Assessments 

Forward  Looking  Infrared  Radar 

Follow-on  Operational  Test  and  Evaluation 

Human-In-The-Loop 

High  Level  Architecture 

Hardware-in-The-Loop 

Initial  Operational  Capability 

Initial  Operational  Test  and  Evaluation 

Integrated  Product  and  Process  Development 

Integrated  Product  Team 

Installed  System  Test  Facility 

Improved  Target  Acquisition  System 

Joint  Advanced  Distributed  Simulation 

Joint  Data  Base  Elements  for  M&S 

Joint  Modeling  And  Simulation  System 

Live  Fire  Test  and  Evaluation 

Low-Rate  Initial  Production 

Limited  User  Test 

Modeling  and  Simulation 
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MNS 

Mission  Need  Statement 

MOE 

Measures  Of  Effectiveness 

MOP 

Measures  Of  Performance 

MRTFB 

Major  Range  and  Test  Facilities  Base 

MSEA 

Modeling  and  Simulation  Executive  Agent 

MSOSA 

M&S  Operational  Support  Activity 

MSRR 

Modeling  and  Simulation  Resource  Repository 

MSTI 

Model-Simulate-T  est-Iterate 

NPS 

Naval  Postgraduate  School 

OA 

Operational  Assessment 

ORD 

Operational  Requirements  Document 

OT 

Operational  Test;  Operational  Testing 

OT&E 

Operational  Test  and  Evaluation 

RAM 

Reliability,  Availability,  and  Maintainability 

RCS 

Radar  Cross  Section 

REP 

Request  For  Proposal 

SAPOR 

Semi-Automated  Forces 

SIL 

System/Software  Integration  Laboratory 

STEP 

Simulation,  Test,  and  Evaluation  Process 

SWIL 

Software-In-The-Loop 

T&E 

Test  and  Evaluation 

TEMP 

Test  and  Evaluation  Master  Plan 

TENA 

Test  and  Training  ENabling  Architecture 

TMD 

Theater  Missile  Defense 

V&V 

Verification  and  Validation 

W&A 

Verification,  Validation,  and  Accreditation 

W&C 

Verification,  Validation,  and  Certification 
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Definitions 


Accreditation.  The  official  certification  that  a  model  or  simulation  is  acceptable  for  use 
for  a  specific  purpose.  [DoDD  5000.59;  DoD  5000. 59-P] 

Architecture.  The  structure  of  components  in  a  program/system,  their  interrelationships, 
and  the  principles  and  guidelines  governing  their  design  and  evolution  over  time.  [DoD 
5000. 59-P] 

Battlespace.  The  physical  environment  in  which  the  simulated  warfare  will  take  place 
and  the  forces  that  will  conduct  the  simulated  warfare.  [DoD  5000. 59-P] 

Computer  Simulation.  A  dynamic  representation  of  a  model,  often  involving  some 
combination  of  executing  code,  control/display  interface  hardware,  and  interfaces  to  real- 
world  equipment.  [DMSO  Glossaiy  of  M&S  Terms] 

Data  Certification.  The  determination  that  data  have  been  verified  and  validated.  Data 
user  certification  is  the  determination  by  the  application  sponsor  or  designated  agent  that 
data  have  been  verified  and  validated  as  appropriate  for  the  specific  M&S  usage.  Data 
producer  certification  is  the  determination  by  the  data  producer  that  data  have  been  veri¬ 
fied  and  validated  against  documented  standards  or  criteria.  [MSMP] 

Data  Validation.  The  documented  assessment  of  data  by  subject  area  experts  and  its 
comparison  to  known  values.  Data  user  validation  is  an  assessment  as  appropriate  for  use 
in  an  intended  model.  Data  producer  validation  is  an  assessment  within  stated  criteria  and 
assumptions.  [DIS;  MSMP] 

Data  Verification.  Data  producer  verification  is  the  use  of  techniques  and  procedures  to 
ensure  that  data  meets  constraints  defined  by  data  standards  and  business  rules  derived 
from  process  and  data  modeling.  Data  user  verification  is  the  use  of  techniques  and  pro¬ 
cedures  to  ensure  that  data  meets  user  specified  constraints  defined  by  data  standards  and 
business  rules  derived  from  process  and  data  modeling,  and  that  data  are  transformed  and 
formatted  properly.  [MSMP] 

Data  Verification,  Validation  &  Certification  (VV&C).  The  process  of  verifying  the 
internal  consistency  and  correctness  of  data,  validating  that  it  represents  real  world  enti¬ 
ties  appropriate  for  its  intended  purpose  or  an  expected  range  of  purposes,  and  certifying 
it  as  having  a  specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of 
use,  or  range  of  uses.  The  process  has  two  perspectives:  producer  and  user  process. 
[MSMP] 

Distributed  Interactive  Simulation  (DIS).  (1)  Program  to  electronically  link  organiza¬ 
tions  operating  in  the  four  domains:  advanced  concepts  and  requirements;  military  opera¬ 
tions;  research,  development,  and  acquisition;  and  training.  (2)  A  synthetic  environment 
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within  which  humans  may  interact  through  simulation(s)  at  multiple  sites  networked  us¬ 
ing  compliant  architecture,  modeling,  protocols,  standards,  and  databases.  [DoD  5000.59- 

P] 

Fidelity.  (1)  The  similarity,  both  physical  and  functional,  between  the  simulation  and  that 
which  it  simulates.  (2)  A  measure  of  the  realism  of  the  simulation.  (3)  The  degree  to 
which  the  representation  within  a  simulation  is  similar  to  a  real  world  object,  feature,  or 
condition  in  a  measurable  or  perceivable  manner.  [DMSO  Glossary  of  M&S  Terms,  DIS 
Glossary] 

High  Level  Architecture  (HLA).  Major  functional  elements,  interfaces,  and  design 
rules,  pertaining  as  feasible  to  all  DoD  simulation  applications,  and  providing  common 
framework  within  which  specific  system  architectures  can  be  defined.  [DMSO  Glossary 
of  M&S  Terms] 

Human-In-The-Loop  (HITL).  A  model  that  requires  human  participation.  [DMSO 
Glossary  of  M&S  Terms;  DIS  Glossary]. 

Measure  of  Effectiveness  (MOE).  A  measure  of  operational  success  that  must  be  closely 
related  to  the  objective  of  the  mission  or  operation  being  evaluated.  Examples  are  kills 
per  shot,  probability  of  kill,  and  effective  range.  A  meaningful  MOE  must  be  quantifiable 
and  a  measure  to  what  degree  the  real  objective  is  achieved. 

Measure  of  Performance  (MOP).  Measure  of  lowest  level  of  performance  representing 
subsets  of  MOEs.  Examples  are  speed,  payload,  range,  time  on  station,  frequency,  or 
other  distinctly  quantifiable  performance  features. 

Model,  n  A  physics,  mathematical,  or  otherwise  logical  representation  of  a  system,  en¬ 
tity,  phenomenon,  or  process.  [DIS  Glossary;  DoD  5000.59-P]  v  To  create  a  representa¬ 
tion  of  a  system,  entity,  phenomenon  or  process. 

Modeling  and  Simulation  (M&S).  The  use  of  models,  including  emulators,  prototypes, 
simulators,  and  stimulators,  either  statically  or  over  time,  to  develop  data  as  a  basis  for 
making  managerial  or  technical  decisions.  The  terms  "modeling"  and  "simulation"  are 
often  used  interchangeably.  [DMSO  Glossaiy  of  M&S  Terms] 

Simulate.  To  represent  a  system  by  a  model  that  behaves  or  operates  like  the  system. 
[DMSO  Glossary  of  M&S  Terms;  DIS  Glossary] 

Simulation.  A  method  for  implementing  a  model  over  time.  [DoDD  5000.59] 

a.  Live  Simulation.  A  simulation  involving  real  people  operating  real  systems. 

b.  Virtual  Simulation.  A  simulation  involving  real  people  operating  simulated  sys¬ 
tems.  Virtual  simulations  inject  HITL  in  a  central  role  by  exercising  motor  control  skills 
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(e.g.,  flying  an  airplane),  decision  skills  (e.g.,  committing  fire  control  resources  to  action), 
or  communications  skills  (e.g.,  as  members  of  a  O'*!  team). 

c.  Constructive  Model  or  Simulation.  Models  or  simulations  that  involve  real  peo¬ 
ple  making  inputs  into  a  simulation  that  carries  out  those  inputs  by  simulated  people  op¬ 
erating  simulated  systems. [  DoD  5000.59-P] 

Simulator.  (1)  A  device,  computer  program,  or  system  that  performs  a  simulation.  (2) 
For  training,  a  device  which  duplicates  the  essential  features  of  a  task  situation  and  pro¬ 
vides  for  direct  practice.  (3)  For  DIS,  a  physical  model  or  simulation  of  a  weapons  sys¬ 
tem,  set  of  weapons  systems,  or  piece  of  equipment  which  represented  some  major  as¬ 
pects  of  the  equipment’s  operation.  [DMSO  Glossary  of  M&S  Terms;  DIS  Glossary] 

Stimulate.  To  provide  input  to  a  system  in  order  to  observe  or  evaluate  the  system’s  re¬ 
sponse.  [DMSO  Glossary  of  M&S  Terms;  DIS  Glossary] 

Synthetic  Environments.  Intemetted  simulations  that  represent  activities  at  a  high  level 
of  realism  from  simulations  of  theaters  of  war  to  factories  and  manufacturing  processes. 
These  environments  may  be  created  within  a  single  computer  or  a  vast  distributed  net¬ 
work  connected  by  local  and  wide  area  networks  and  augmented  by  super-realistic  special 
effects  and  accurate  behavioral  models.  They  allow  complete  visualization  of  and  total 
immersion  into  the  environment  being  simulated.  [DoD  5000.59-P] 

Validation.  The  process  of  determining  the  degree  to  which  a  model  or  simulation  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the 
model  or  simulation.  [DoDD  5000.59] 

Verification.  The  process  of  determining  that  a  model  or  simulation  implementation  ac¬ 
curately  represents  the  developer’s  conceptual  description  and  specification.  Verification 
also  evaluates  the  extent  to  which  the  model  or  simulation  has  been  developed  using 
sound  and  established  software  engineering  techniques.  [DoDD  5000.59] 

Virtual  Prototype.  A  computer-based  simulation  of  systems  and  subsystems  which  ex¬ 
hibits  both  geometric  and  functional  realism.  This  three-dimensional  virtual  mockup  may 
be  used  to  evaluate  prototypes  or  concepts  and  provides  a  common  platform  which  all 
functional  disciplines  (design,  test,  manufacturing,  logistics,  training,  operations,  etc.)  can 
work.  [DMSO  Glossary  of  M&S  Terms;  DSMC  Guidebook  on  M&S] 

Virtual  Prototyping.  The  process  of  using  a  virtual  prototype,  in  lieu  of  a  physical  pro¬ 
totype,  for  test  and  evaluation  of  specific  characteristics  of  a  candidate  design.  [DSMC 
Virtual  Prototyping] 
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Some  Helpful  Internet  Sites 


Site 

DTSE&E 

•  STEP  Guidelines 
DOT&E 

DD,  LFT&E 
DMSO 

•  DoD  M&S  Master  Plan 

•  HLA 

•  VV&A 
DMSTTIAC 
JADS  JTF 
JITC 
JMASS 

JSIMS  Joint  Program  Office 
JWARS  Office 
MSOSA 
MSRR 

Software  Technology  Support  Center 
TENA 

AF  Directorate  for  M&S  and  Analysis 

AF  M&S  Resource  Repository 

AFMC  M&S  TPIPT 

AFOTEC 

Army  OPTEC 

Army  M&S  Office 

•  Army  M&S  Master  PI  an 


URL  Address 

http://www.acq.osd.mil/te/ 

http://www.acq.osd.niil/te/programs/tfr/step.htm 

http://www.dote.osd.mil/ 

http :  //www.  dote .  osd .  mil/lfte/INDEX.HTML 

http://www.dmso.mil/ 

http://www.dmso.mi1/docslib/#mspolicy/ 

http://www.dmso.mil/proj  ects/hla/ 

http://www.dmso.mil/proj  ects/vva/ 

http :  //  dmsttiac .  hq.iitri .  com/ 

http  ://j  adsweb .  kirtl  and .  af  mil/ 

http  ://138.27.8. 2/ot&  e/otande .  htm 

http://www.jmass.wpafb.af.mil/ 

http://www.j  sims.mil/ 

http://www.dtic.mil/defenselink/jwars/ 

http://www.msosa.mil.inter.net/ 

http://mercuiy-www4.nosc.mil/msrr/ 

http://stscols.hill.af  mil/ 

http  ://c3  8 .  npt.nuwc .  navy  .mil/TENA/home.html 

http://xom.hq.af.mil/ 

http://afmsrr.sc.ist.ucf  edu/ 

http://www.afbmd.laafb.af.mil/xre/m&s/ 

http : //www.  afotec.  af  mil/ 

http:// 

http://www.misma.army.mil/ 

http://www.misma.army.mil/ 


Army  M&S  Resource  Repository  http://www.misma.army.mil/armymsrr/ 


M&S  Army  Integrated  Catalog 
Navy  M&S  Management  Office 
Navy  M&S  Resource  Repository 
Navy  COMOPTEVFOR 
Navy  T&E  M&S  Management  Office 
Navy  T&E  Repository  for  M&S 
USMC  M&S  Management  Office 


http://hpO  1  .arc.iquest.com/mosaic/mosaic.html 

http  ://sneezy. nosc.mil :  80/donmsmo/navmsmo/ 

http://navmsmo.nosc.mil/msrr/ 

http  ://tecnet0.j  cte.j  cs  .mil :  9000/COTFOTD/ 

http://www.nawcad.navy.mil/tems/index.html 

http  ://nterms.mugu.navy  .mil/ 

http://mcmsmo.usmc.mil/ 


blank  page 
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